webscraping.app
Product
Browse
Guides
Company
Sign In
Get early access
Product
Features
Pricing
Browse
Categories
Crawler checker
Tools
Guides
AI Scraping
No-Code
Open Source
Scraping APIs
Company
About Us
Contact us
Submit a tool
AI crawler and robots.txt checker
Check which search and AI crawlers can access your site based on its robots.txt rules.
FAQ
Questions, answered
How it works
What exactly does this tool check?
Where does your AI crawler list come from?
Do you store the domains or results I check?
Basics
What is robots.txt, and do bots have to obey it?
What are Google-Extended and TDM-Reservation?
How is blocking a bot in robots.txt different from blocking it at the network layer?
Scope & legal
Does this result tell me whether scraping a site is legal?
What can and can't I use this tool for?
The result says a bot is "allowed" - can I scrape?