Apache Hudi
Apache Hudi is an open data lakehouse platform that enables efficient record-level upserts, incremental processing, and ACID transactions on data lakes.
Apache Hudi is a powerful, open-source lakehouse platform that reimagines batch processing with efficient incremental data pipelines.
Key Features:
- Record-Level Upserts - Quickly update and delete individual records with fast, pluggable indexing.
- ACID Transactions - Guarantee atomic writes with snapshot isolation tailored for lake-scale operations.
- Time Travel - Query historical data, audit changes, and roll back to previous table versions.
- Multi-Engine Integration - Works with Spark, Flink, Presto, Trino, Hive, and dbt orchestration.
- Streaming Ingestion - Ingest from Kafka, Pulsar, and CDC sources with built-in deduplication.
Whether you're building real-time analytics, managing CDC pipelines, or modernizing batch ETL, Apache Hudi delivers efficient lakehouse storage with minute-level freshness.
Web scraping workflow
How to use Apache Hudi in a web scraping pipeline
Apache Hudi adds table-management capabilities, including upserts and deletes, to data-lake storage. It can organize repeatedly scraped entities when new observations must update an existing analytical table.
A practical workflow
- A separate scraper exports validated records with stable entity keys, source URLs and observation times.
- A compatible processing engine writes those records into Hudi tables using an explicit key and update-order policy.
- A supported query engine reads the resulting table for analysis, while the pipeline retains references to original captures.
When it fits
Consider it when a file-based data lake needs incremental updates rather than a new disconnected export after every crawl. The record identity and ordering policy should reflect the difference between current state and retained historical observations.
Boundaries
Hudi is not a website fetcher or a standalone SQL database service. Storage, compatible compute and table maintenance must be supplied and configured. Upsert support does not decide which conflicting observation is correct; the loader still needs rules for delayed runs, missing fields and changed entity identifiers.
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Our productRepository
apache/hudi
Stars
6.3KForks
2.5KLast commit
2026-10-15License
Apache-2.0Language
Java
Metrics recorded 2026-10-16
Repository details
License source: Repository license file
Frequently asked questions
What is Apache Hudi used for?
Apache Hudi manages transactional tables on data-lake storage, including updates, deletes, indexes and incremental processing. Compatible engines read and write those tables. Its ingestion and table-maintenance services support pipelines that update datasets over time.
Sources (checked ): Apache Hudi overview
How does Hudi work?
Hudi tracks table actions in a timeline and organizes data into file groups and file slices. Copy-on-Write stores table state in base files; Merge-on-Read combines base files with change logs. Readers use the appropriate query type, and compaction can merge logged changes into base files.
Sources (checked ): Hudi current storage specification; Hudi table types
What file format does Hudi use?
Hudi Technical Specification 1.0, updated August 2026 for table version 9 and reflecting Hudi 1.2.0, lists Parquet, ORC, HFile and Lance base files. Lance is supported for base-file reads and Copy-on-Write writes only; column and partition statistics are not populated for Lance files. Log records can use Avro, Parquet or HFile. Support in a specific reader or writer depends on its engine, version and configuration.
Sources (checked ): Hudi current storage specification
What are the limitations of Apache Hudi?
Copy-on-Write updates can rewrite base files, while Merge-on-Read shifts work toward log merging and compaction. Table maintenance, indexing and engine compatibility require configuration and capacity planning. Supported operations depend on the table type, storage format and reader/writer versions.
Sources (checked ): Hudi table types; Hudi current storage specification
What are the key differences between Apache Hudi and Apache Iceberg?
Both provide table management over data-lake storage. Hudi documents incremental processing, upserts and Copy-on-Write or Merge-on-Read table types. Iceberg has its own snapshot, metadata and engine-integration model. Compare update patterns, maintenance and reader/writer compatibility for your platform.
Sources (checked ): Apache Hudi overview; Hudi table types; Apache Iceberg documentation
Sources
Verified against the apache/hudi repository and the PyPI package.