Data Processing

Quick answer

Data processing choices depend on what happens after collection. Use workflow orchestration for schedules and dependencies, task queues for background jobs, ETL tools for movement between systems, transformation tools for cleaning and validation, distributed crawling for collection at scale, and document extraction for PDFs, Office files, and scanned content.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
981
Forks
282
Last commit
21 days ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Unstructured

Clean Documents For AI Pipelines

Stars
15K
Forks
1.3K
Last commit
22 days ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Apache Tika

Extract Text From Any Document

Stars
3.8K
Forks
941
Last commit
21 days ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
62.2K
Forks
4.4K
Last commit
21 days ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last commit
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last commit
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.3K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Nutch

Highly extensible and scalable web crawler

Stars
3.1K
Forks
1.3K
Last commit
5 months ago

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Heritrix

Web-scale archival web crawler

Stars
3.2K
Forks
782
Last commit
6 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Inngest

Durable Functions for Workflows and Queues

Stars
4.7K
Forks
240
Last commit
5 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

dlt (data load tool)

Python Library for Data Loading

Stars
4.8K
Forks
0

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last commit
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Huey

Lightweight Python Task Queue

Stars
5.9K
Forks
392
Last commit
6 months ago

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

Mage AI

AI-Powered Data Pipeline Builder

Stars
8.6K
Forks
905
Last commit
5 months ago

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

RQ (Redis Queue)

Simple Job Queues for Python

Stars
10.6K
Forks
1.5K
Last commit
5 months ago

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt

Transform Data in Your Warehouse with SQL

Stars
10.9K
Forks
0

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations

Data Quality Testing and Validation Platform

Stars
11.1K
Forks
0

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

Dagster

Unified Platform for AI and Data Pipelines

Stars
14.8K
Forks
2K
Last commit
5 months ago

Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.

Windmill

Build and Deploy Internal Tools at Scale

Stars
15.6K
Forks
869
Last commit
5 months ago

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Temporal

Durable Execution for Invincible Applications

Stars
17.9K
Forks
1.3K
Last commit
5 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Luigi

Python Batch Pipeline Framework by Spotify

Stars
18.6K
Forks
2.4K
Last commit
5 months ago

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Airbyte

Open-Source Data Integration Platform

Stars
20.5K
Forks
0

Airbyte is an open-source ELT platform with 600+ connectors that moves data from APIs, databases, and files to warehouses and data lakes at scale.

Prefect

Workflow Orchestration for Python Data Pipelines

Stars
21.4K
Forks
2.1K
Last commit
5 months ago

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Kestra

Declarative Data Orchestration Platform

Stars
26.3K
Forks
2.5K
Last commit
5 months ago

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Pydantic

Data Validation Using Python Type Hints

Stars
26.5K
Forks
0

Pydantic is the most widely used Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Celery

Distributed Task Queue for Python

Stars
27.9K
Forks
4.9K
Last commit
5 months ago

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Polars

Lightning-Fast DataFrame Library in Rust

Stars
37.1K
Forks
0

Polars is an open-source, Rust-powered DataFrame library delivering 10-100x faster data processing than pandas with lazy evaluation and parallelism.

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Stars
44K
Forks
16.3K
Last commit
5 months ago

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

Scrapy

Fast, high-level web crawling framework for Python

Stars
59.5K
Forks
11.2K
Last commit
5 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

n8n

AI Workflow Automation for Technical Teams

Stars
170.9K
Forks
54K
Last commit
5 months ago

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Apache StormCrawler

Scalable Java Crawlers For Teams

Stars
981
Forks
282
Last commit
21 days ago

Build scalable Java crawlers on Apache Storm with low-latency processing, flexible customization, and an Apache-2.0 license.

Unstructured

Clean Documents For AI Pipelines

Stars
15K
Forks
1.3K
Last commit
22 days ago

An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.

Apache Tika

Extract Text From Any Document

Stars
3.8K
Forks
941
Last commit
21 days ago

Detects file types and extracts text and document details from 1,000+ formats through one Java interface.

Docling

Turn Documents Into AI-Ready Data

Stars
62.2K
Forks
4.4K
Last commit
21 days ago

Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.

AWS Athena

Serverless SQL Queries on Amazon S3

ETL Tools

Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.

AWS Glue

Serverless Data Integration Service by AWS

ETL Tools

AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.

Scrapy Cluster

Distributed on-demand scraping with Scrapy

Stars
1.2K
Forks
322
Last commit
2 years ago

Scrapy Cluster uses Redis and Kafka to create a distributed, on-demand Scrapy crawling cluster for coordinated large-scale web scraping.

Frontera

Scalable crawl frontier framework

Stars
1.3K
Forks
217
Last commit
last year

Frontera is a Python crawl frontier framework for managing when and what to crawl, enabling web crawlers of any scale with Scrapy integration.

Meltano

CLI-First ELT Engine Built on Singer

Stars
2.3K
Forks
0

Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.

Nutch

Highly extensible and scalable web crawler

Stars
3.1K
Forks
1.3K
Last commit
5 months ago

Apache Nutch is a highly extensible, production-ready web crawler built on Hadoop for large-scale batch crawling and data acquisition tasks.

Heritrix

Web-scale archival web crawler

Stars
3.2K
Forks
782
Last commit
6 months ago

Heritrix is the Internet Archive's open-source, extensible, archival-quality web crawler designed for large-scale web preservation and data collection.

Inngest

Durable Functions for Workflows and Queues

Stars
4.7K
Forks
240
Last commit
5 months ago

Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.

dlt (data load tool)

Python Library for Data Loading

Stars
4.8K
Forks
0

dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.

Scrapy Redis

Redis-based distributed components for Scrapy

Stars
5.6K
Forks
1.6K
Last commit
2 years ago

Scrapy-Redis provides Redis-backed components for Scrapy, enabling distributed crawling with shared request queues and item pipelines.

Huey

Lightweight Python Task Queue

Stars
5.9K
Forks
392
Last commit
6 months ago

Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.

BullMQ

Message Queue and Background Jobs for Node.js

Stars
8.3K
Forks
547
Last commit
5 months ago

BullMQ is an open-source Redis-based message queue trusted by thousands of companies processing billions of jobs daily across Node.js, Python, and more.

Mage AI

AI-Powered Data Pipeline Builder

Stars
8.6K
Forks
905
Last commit
5 months ago

Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.

RQ (Redis Queue)

Simple Job Queues for Python

Stars
10.6K
Forks
1.5K
Last commit
5 months ago

RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.

dbt

Transform Data in Your Warehouse with SQL

Stars
10.9K
Forks
0

dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.

Great Expectations

Data Quality Testing and Validation Platform

Stars
11.1K
Forks
0

Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.

Dagster

Unified Platform for AI and Data Pipelines

Stars
14.8K
Forks
2K
Last commit
5 months ago

Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.

Windmill

Build and Deploy Internal Tools at Scale

Stars
15.6K
Forks
869
Last commit
5 months ago

Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.

Temporal

Durable Execution for Invincible Applications

Stars
17.9K
Forks
1.3K
Last commit
5 months ago

Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.

Luigi

Python Batch Pipeline Framework by Spotify

Stars
18.6K
Forks
2.4K
Last commit
5 months ago

Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.

Airbyte

Open-Source Data Integration Platform

Stars
20.5K
Forks
0

Airbyte is an open-source ELT platform with 600+ connectors that moves data from APIs, databases, and files to warehouses and data lakes at scale.

Prefect

Workflow Orchestration for Python Data Pipelines

Stars
21.4K
Forks
2.1K
Last commit
5 months ago

Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.

Kestra

Declarative Data Orchestration Platform

Stars
26.3K
Forks
2.5K
Last commit
5 months ago

Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.

Pydantic

Data Validation Using Python Type Hints

Stars
26.5K
Forks
0

Pydantic is the most widely used Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.

Celery

Distributed Task Queue for Python

Stars
27.9K
Forks
4.9K
Last commit
5 months ago

Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.

Polars

Lightning-Fast DataFrame Library in Rust

Stars
37.1K
Forks
0

Polars is an open-source, Rust-powered DataFrame library delivering 10-100x faster data processing than pandas with lazy evaluation and parallelism.

Apache Airflow

Programmatically Author, Schedule, and Monitor Workflows

Stars
44K
Forks
16.3K
Last commit
5 months ago

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.

Scrapy

Fast, high-level web crawling framework for Python

Stars
59.5K
Forks
11.2K
Last commit
5 months ago

Scrapy is an open-source Python framework for building fast, scalable web crawlers that extract structured data from websites efficiently.

n8n

AI Workflow Automation for Technical Teams

Stars
170.9K
Forks
54K
Last commit
5 months ago

n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.

Browse 6 subcategories of Data Processing tools and find the perfect solution for your needs.

Quick answer

Related categories