Data Processing

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping.

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Airbyte is an open-source ELT platform with 600+ connectors that moves data from APIs, databases, and files to warehouses and data lakes at scale.

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Pydantic is the most widely used Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Polars is an open-source, Rust-powered DataFrame library delivering 10-100x faster data processing than pandas with lazy evaluation and parallelism.

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.