Data Processing

Serverless SQL queries on S3. Query scraped data files directly without loading. Pay per query.

Serverless ETL service. Data catalog, crawlers, Spark-based transforms. Native S3 and Redshift integration.

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

A scalable frontier for web crawlers

Code-first ELT engine built on Singer. GitOps workflows, 300+ connectors, integrates with dbt.

Apache Nutch is an extensible and scalable web crawler

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

The leading workflow orchestration platform. Run stateful step functions and AI workflows on serverless, servers, or the edge.

Lightweight Python library for data loading. Auto schema inference, 5000+ sources supported.

Redis-based components for Scrapy.

Tiny but feature-rich, Redis or SQLite

BullMQ - Message Queue and Batch processing for NodeJS, Python, Elixir and PHP based on Redis

Hybrid Python/SQL/R in same pipeline

Simple, lightweight, Redis-based

Transform data in your warehouse using SQL. Version control, testing, documentation for data models.

Data quality testing with 'Expectations'. Validate scraped data, auto-generate docs, CI/CD integration.

An orchestration platform for the development, production, and observation of data assets.

Fast execution, code + visual builder

Temporal service

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Open-source ELT platform with 600+ connectors. Move data from APIs, databases, files to warehouses and lakes.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Data validation using Python type hints. Rust-powered core for speed. Define schemas for scraped data.

Distributed Task Queue (development branch)

Lightning-fast DataFrame library in Rust. 10-100x faster than pandas. Lazy evaluation, out-of-core processing.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Scrapy, a fast high-level web crawling & scraping framework for Python.

Low-code/no-code, larger community