Clean Documents For AI Pipelines
Library or tool · Self-hosted
An open-source Python document ETL tool that turns PDFs, HTML, images, and more into structured elements for RAG and LLM workflows.
Document Extraction
Extract Text From Any Document
Library or tool · Self-hosted
Detects file types and extracts text and document details from 1,000+ formats through one Java interface.
Document Extraction
Turn Documents Into AI-Ready Data
Library or tool · Self-hosted
Open-source Python software that converts PDFs, DOCX, PPTX, and images into structured data for GenAI and RAG pipelines.
Document Extraction
Serverless SQL Queries on Amazon S3
Platform or service · API
Amazon Athena is a serverless, interactive query service that lets you analyze petabyte-scale data in S3 using standard SQL with pay-per-query pricing.
Data Processing
Serverless Data Integration Service by AWS
Platform or service · API
AWS Glue is a fully managed, serverless ETL service that discovers, prepares, and integrates data from over 100 sources into lakes and warehouses.
ETL Tools
CLI-First ELT Engine Built on Singer
Library or tool · Self-hosted
Meltano is an open-source, CLI-first ELT platform with 600+ connectors, GitOps workflows, and native dbt integration for data pipeline management.
ETL Tools
Durable Functions for Workflows and Queues
Library or tool · Self-hosted
Inngest replaces traditional queues with durable step functions, enabling reliable multi-step workflows on serverless, servers, or the edge.
Workflow Orchestration
Python Library for Data Loading
Library or tool · Self-hosted
dlt is a lightweight, open-source Python library for loading data from any source into well-structured datasets with automatic schema inference.
Data Transformation & Validation+1 more
Lightweight Python Task Queue
Library or tool · Self-hosted
Huey is a lightweight Python task queue with a clean API supporting Redis, SQLite, or in-memory storage for background job processing.
Task Queues
Message Queue and Background Jobs for Node.js
Library or tool · Self-hosted
BullMQ is a task queue for background jobs. Its JavaScript/TypeScript package supports Redis; version 6.3.11 also includes an optional PostgreSQL backend. Backend support must be checked for each language binding.
Task Queues
AI-Powered Data Pipeline Builder
Library or tool · Self-hosted
Mage AI is a data engineering platform that lets you build, deploy, and run scalable data pipelines with AI assistance across Python, SQL, and R.
Workflow Orchestration
Simple Job Queues for Python
Library or tool · Self-hosted
RQ is a lightweight Python library for queueing jobs and processing them in the background with Redis-backed workers.
Task Queues
Transform Data in Your Warehouse with SQL
Library or tool · Self-hosted
dbt empowers data teams to transform, test, and document data directly in the warehouse using SQL, with version control and governance built in.
Data Transformation & Validation
Data Quality Testing and Validation Platform
Library or tool · Self-hosted
Great Expectations helps data teams validate, document, and monitor data quality across pipelines with automated testing and observability.
Data Transformation & Validation
Unified Platform for AI and Data Pipelines
Library or tool · Self-hosted
Dagster is a data orchestration platform that helps teams build, schedule, and monitor reliable data pipelines with integrated lineage and observability.
Workflow Orchestration
Build and Deploy Internal Tools at Scale
Library or tool · Self-hosted
Windmill is an open-source platform for building workflows, internal tools, and data pipelines with full code flexibility and a visual flow builder.
Workflow Orchestration
Durable Execution for Invincible Applications
Library or tool · Self-hosted
Temporal is a durable execution platform that makes distributed applications fault-tolerant by automatically capturing state and recovering from failures.
Workflow Orchestration
Python Batch Pipeline Framework by Spotify
Library or tool · Self-hosted
Luigi is a Python package by Spotify for building complex pipelines of batch jobs with dependency resolution, workflow management, and Hadoop support.
Workflow Orchestration
Data Integration Platform with Source-Available Core
Platform or service · API / SaaS / Self-hosted
Airbyte is a data integration platform with a source-available core and connectors for moving data into warehouses and data lakes.
ETL Tools
Workflow Orchestration for Python Data Pipelines
Library or tool · Self-hosted
Prefect is an open-source Python workflow orchestration framework for building resilient data pipelines with full observability and zero rewrites.
Workflow Orchestration
Declarative Data Orchestration Platform
Library or tool · Self-hosted
Kestra is an open-source orchestration platform that uses declarative YAML to build, schedule, and monitor event-driven workflows at scale.
Workflow Orchestration
Data Validation Using Python Type Hints
Library or tool · Self-hosted
Pydantic is a Python data validation library, offering fast schema enforcement with type hints and a Rust-powered core engine.
Data Transformation & Validation
Distributed Task Queue for Python
Library or tool · Self-hosted
Celery is a simple, flexible, and reliable distributed task queue for processing vast amounts of messages with real-time and scheduled execution.
Task Queues
Lightning-Fast DataFrame Library in Rust
Library or tool · Self-hosted
Polars is an open-source DataFrame library with a Rust-based query engine, columnar processing, lazy evaluation, and parallel execution.
Data Transformation & Validation
Programmatically Author, Schedule, and Monitor Workflows
Library or tool · Self-hosted
Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring data pipelines and batch workflows.
Workflow Orchestration
AI Workflow Automation for Technical Teams
Platform or service · API / SaaS / Self-hosted
n8n is a workflow automation platform that combines AI capabilities with business process automation, offering code flexibility with no-code speed.
Workflow Orchestration
