Web Scraping Tools

Apache Nutch is an extensible and scalable web crawler

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Successor to undetected-chromedriver. CDP-based browser automation with built-in anti-detection. No Selenium dependency.

HTML5 spec compliant parser

Managed search, great for e-commerce

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

Low-level, fast, forgiving HTML parser

Lightweight Python library for data loading. Auto schema inference, 5000+ sources supported.

A Python library for automating interaction with websites.

Anti-detect Firefox browser with fingerprint injection & anti-bot evasion. Playwright-compatible.

Redis-based components for Scrapy.

Tiny but feature-rich, Redis or SQLite

Lakehouse platform for upserts and incremental processing. Efficient record-level updates.

TypeScript library for structured data extraction

DevTools Protocol, handles JavaScript

AWS managed, auto-scaling NoSQL

BullMQ - Message Queue and Batch processing for NodeJS, Python, Elixir and PHP based on Redis

Open table format for huge analytics tables. Multi-engine (Spark, Trino, Flink). Used by Netflix, Airbnb.

Lakehouse storage framework. ACID transactions, schema evolution, time travel. Databricks standard.

Serverless vector database in Rust. Embedded or cloud, no server needed. Native DataFrame integration.

Hybrid Python/SQL/R in same pipeline

Adaptive scraping that learns from website changes

High availability, peer-to-peer architecture

Node.js automation framework with extensive plugin ecosystem

WebDriver-free browser testing, no plugins needed

Simple, lightweight, Redis-based

Transform data in your warehouse using SQL. Version control, testing, documentation for data models.

Data quality testing with 'Expectations'. Validate scraped data, auto-generate docs, CI/CD integration.

Faster joins, easier maintenance

2.8x faster for big data, 10x faster log analytics

AWS-backed fork, Apache 2.0 license

Deploy headless browsers in Docker. Run on our cloud or bring your own. Free for non-commercial uses.

Multithreaded, 1M+ ops/sec per node

Headless Chrome automation in Go

Real-time OLAP database for event-driven data. Sub-second queries, streaming ingestion.

Lightweight in-memory caching, multi-threaded