Apache Iceberg
Apache Iceberg is a high-performance open table format enabling multi-engine analytics with schema evolution, time travel, and hidden partitioning.
Apache Iceberg is a high-performance, open table format that brings the reliability of SQL tables to petabyte-scale analytic datasets.
Key Features:
- Multi-Engine Support - Query the same tables safely from Spark, Trino, Flink, Presto, and Hive simultaneously.
- Hidden Partitioning - Skip unnecessary partitions automatically without extra query filters.
- Schema Evolution - Add, rename, and reorder columns without rewriting your tables.
- Time Travel - Query historical snapshots and roll back to previous table versions instantly.
- Expressive SQL - Merge, update, and delete rows with flexible SQL commands and delta support.
- Industry Adopted - Trusted by Netflix, Airbnb, and thousands of production environments.
Whether you're building a data lakehouse, running large-scale analytics, or unifying batch and stream pipelines, Apache Iceberg provides a reliable open table format.
Web scraping workflow
How to use Apache Iceberg in a web scraping pipeline
Apache Iceberg is an open table format used by compatible processing and query engines. It can organize scraped datasets as versioned analytical tables rather than a collection of loosely related export files.
A practical workflow
- A crawler saves validated structured records and a manifest linking them to their capture run.
- A compatible writer commits those records into an Iceberg table using the configured catalog and storage system.
- Query a selected table snapshot to reproduce an analysis or compare collection batches without confusing file discovery with table state.
When it fits
Use it when a data lake needs table snapshots, schema evolution and access through supported engines. Preserving the source-run identifier makes an analytical result traceable even when the table's schema later changes.
Boundaries
Iceberg is a table format, not an HTML scraper or a complete database runtime. You still need storage, catalog configuration and compatible compute. Snapshot retention and file cleanup require planning, and table-level history does not replace keeping the raw source evidence needed to investigate an extraction error.
Growing a website? Track its keywords, Google rankings, Search Console insights, and share of voice with bisibility. Free during beta.
Our productRepository
apache/iceberg
Stars
9.3KForks
3.6KLast commit
2026-10-15License
Apache-2.0Language
Java
Metrics recorded 2026-10-16
Repository details
License source: Repository license file
Frequently asked questions
What are the uses of Apache Iceberg?
Apache Iceberg is a table format for large analytical datasets. It provides table metadata, snapshots and schema and partition evolution so compatible engines can read and update tables consistently. It is a table layer rather than a standalone query engine.
Sources (checked ): Apache Iceberg documentation
What is Apache Iceberg vs Parquet?
Parquet is a columnar file format. Iceberg defines table metadata and tracks the data files and snapshots that make up a table; Parquet can be one of those file formats.
Sources (checked ): Apache Iceberg documentation; Apache Parquet overview
What are the key differences between Apache Iceberg and Delta Lake table formats?
Both provide transactions and versioned table metadata over data-lake storage. They have different protocols, metadata structures and engine integrations. Choose according to the readers, writers and catalog support required by your data platform.
Sources (checked ): Apache Iceberg documentation; Delta Lake introduction
What are the key differences between Apache Hive and Apache Iceberg?
Hive includes a query system and metastore ecosystem; Iceberg defines table metadata and operations for multiple engines. Iceberg’s schema and partition evolution differ from conventional Hive table layouts.
Sources (checked ): Apache Iceberg documentation; Apache Hive documentation
What are the key differences between Apache Iceberg and Apache Spark?
Iceberg defines how a table’s data and metadata are organized and updated. Spark is a data-processing engine that can read and write Iceberg through its integration. They are commonly complementary components.
Sources (checked ): Apache Iceberg documentation; Apache Spark documentation
Can Databricks use Apache Iceberg tables?
Yes. Databricks documents managed Iceberg tables and support for external Iceberg access patterns. Features depend on the table type, catalog and runtime; follow the current supported configuration for your deployment.
Sources (checked ): Databricks Iceberg support; Apache Iceberg documentation
What are some alternatives to Apache Iceberg?
Delta Lake and Apache Hudi are other table/storage projects for analytical data lakes. Compare the transaction model, update operations and the catalogs and engines that must access your tables.
Sources (checked ): Apache Iceberg documentation; Delta Lake introduction; Apache Hudi documentation
Sources
Verified against the apache/iceberg repository and the PyPI package.