webscraping.app
Tools
Categories
Alternatives
Guides
Free tools
Sign In
Submit a tool
Browse
Tools
Categories
Alternatives
Guides
AI scraping
No-code scraping
Open-source tools
Scraping APIs
Sitemap scrapers
Free tools
Crawler checker
Sitemap Explorer
Submit a tool
Sign In
bisibility
— Track Google rankings and Search Console insights. Free during beta.
Try bisibility free
Our product
AI crawler and robots.txt checker
Check the published robots.txt rules for search and AI crawlers. These rules do not confirm actual network access.
FAQ
Questions, answered
How it works
What exactly does this tool check?
Where does your AI crawler list come from?
Do you store the domains or results I check?
Basics
What is robots.txt, and do bots have to obey it?
What are Google-Extended and TDM-Reservation?
How is blocking a bot in robots.txt different from blocking it at the network layer?
Scope & legal
Does this result tell me whether scraping a site is legal?
What can and can't I use this tool for?
The result says a bot is "allowed" - can I scrape?
Site or robots.txt URL
Check
Try
vercel.com
nytimes.com
github.com