Crawlee

Crawlee is a versatile, developer-friendly web scraping library for JavaScript and Python that handles the hard parts of crawling automatically.

Key Features:

• Anti-Blocking - Automatic proxy rotation, browser fingerprinting, and retry strategies built in
• Multi-Backend - Use Playwright, Puppeteer, Cheerio, or raw HTTP depending on your needs
• Autoscaling - Dynamically adjusts concurrency based on available system resources
• Request Queue - Persistent crawl queue with automatic deduplication and error handling
• Storage - Built-in dataset and key-value store for scraped data management
• TypeScript First - Full TypeScript support with excellent type definitions

Whether you're extracting data for AI training, building monitoring tools, or automating research, Crawlee provides a reliable foundation that scales.