webscraping.app
Features
Pricing
Sign In
Features
Pricing
AI crawler and robots.txt checker
Check which search and AI crawlers can access your site based on its robots.txt rules.
FAQ
Questions, answered
How it works
What exactly does this tool check?
Where does your AI crawler list come from?
Do you store the domains or results I check?
Basics
What is robots.txt, and do bots have to obey it?
What are Google-Extended and TDM-Reservation?
How is blocking a bot in robots.txt different from blocking it at the network layer?
Scope & legal
Does this result tell me whether scraping a site is legal?
What can and can't I use this tool for?
The result says a bot is "allowed" - can I scrape?