Crawlee

Crawlee is a versatile, developer-friendly web scraping library for JavaScript and Python that handles the hard parts of crawling automatically.

Key Features:

• Anti-Blocking — Automatic proxy rotation, browser fingerprinting, and retry strategies built in
• Multi-Backend — Use Playwright, Puppeteer, Cheerio, or raw HTTP depending on your needs
• Autoscaling — Dynamically adjusts concurrency based on available system resources
• Request Queue — Persistent crawl queue with automatic deduplication and error handling
• Storage — Built-in dataset and key-value store for scraped data management
• TypeScript First — Full TypeScript support with excellent type definitions

Whether you're extracting data for AI training, building monitoring tools, or automating research, Crawlee provides a reliable foundation that scales.