Apache Iceberg Adopt

data-platformopen-sourcegovernanceapache-icebergtable-formatlakehousecatalog-governancemulti-engine-interoperabilitytable-maintenanceai-ml-data

Overview

Apache Iceberg is an open table format for large analytic datasets that brings SQL-table semantics — snapshots, schema and partition evolution, row-level deletes — to object storage, so multiple engines can safely operate on the same tables. That framing is now uncontroversial. Iceberg has become the de-facto standard across Spark, Trino, Flink and DuckDB, and by mid-2026 it functions as the interchange layer for the read path: Delta publishes Iceberg metadata synchronously via UniForm, Paimon registers tables into Iceberg REST catalogs, and enterprise engines read Iceberg metadata natively (llms3, tech-insider).

Iceberg stays in adopt, and the reason has shifted from capability to gravity. At Iceberg Summit 2026, over 600 attendees across 70+ sessions, and by the organisers' own account not a single talk tried to convince anyone to adopt Iceberg — every session assumed you were already running it and asked what comes next (Snowflake Engineering). Enterprise survey results point the same way, with over 80% of surveyed organisations using Iceberg to run AI and ML workloads (State of Apache Iceberg in the Enterprise 2026).

The consequence for this radar is that the interesting decisions have moved up the stack. As one market read puts it, Iceberg is the table format layer of the open lakehouse and the fight moved elsewhere: to the catalog, to table maintenance, to who governs access when the caller is an agent rather than a person, and to which parts of the stack stay open once a vendor manages them for you (Iceberg Lakehouse). This entry therefore focuses on operating and scaling Iceberg rather than justifying it.

Adoption Signals

  • The current stable line is Apache Iceberg 1.11.0, released 19 May 2026, which makes Spark 4.1 and Flink 2.1 the default build targets, drops Java 11 support, and deprecates position delete files with row data (Apache Iceberg Releases, Google Open Source Blog).
  • 1.11.0 turns V3 from experimental into production-ready: deletion vectors backed by Roaring bitmaps replace fragmented positional delete files, a native Variant type handles semi-structured data, the REST catalog gains server-side scan planning, and built-in table encryption arrives with envelope encryption and KMS support alongside a pluggable File Format API (LakeOps, Google Open Source Blog).
  • Grab is transitioning a data lake of petabytes across billions of S3 objects from directory-based Hive Parquet and Hive Metastore to a table-centric Iceberg architecture, driving both its batch transformation platform (Slide) and its online-to-lake ingestion platform (Hugo), and is open-sourcing a UnifiedSparkCatalog that hides table-format differences from users (Grab Engineering).
  • Warehouse vendors have moved from read-only tolerance to lifecycle ownership: Snowflake elevated Iceberg to a first-class table format with full lifecycle management and automatic compaction, with write support for externally managed Iceberg tables reaching general availability in October 2025 (AlgeriaTech).
  • Databricks now presents a dual-format lakehouse supporting both Delta Lake and Apache Iceberg alongside hybrid approaches such as UniForm, making format choice an architectural decision shaped by cost efficiency, governance and long-term ecosystem flexibility rather than a platform constraint (Sigmoid, BigDataBoutique).
  • Practitioner accounts have shifted from evaluation to steady-state operations, with published thresholds and patterns for keeping Iceberg lakehouses running without query planning degrading in production (Mohadata).

Risks

  • Governance is fragmented across engines, and agents make it worse. Iceberg makes shared storage portable across Spark, Trino, Snowflake and AI workloads, but the practical problem is fragmented governance, inconsistent masking and uncontrolled agent access; portability without centralised policy enforcement expands exposure rather than reducing it (NHI Mgmt, policy drift).
  • Identity consistency becomes a cross-platform problem. The same dataset may be reachable by users, service accounts and AI agents via different control paths, so a principal restricted in one engine can be effectively privileged in another unless access decisions are centralised (NHI Mgmt FAQ).
  • Secure multi-engine access remains unsolved in the open. An Apache dev-list thread states plainly that there is no easy way to secure Iceberg data, and that secure access across multiple tools and engines is one of the main obstacles to integration; the project's own security model notes Iceberg is a format and library set embedded in larger systems, mostly reflecting the trust decisions of catalogs, engines and operators (Apache dev list, Apache Iceberg Security).
  • V3 engine support is uneven and migration-order sensitive. Databricks has deletion vectors generally available but V2 reads break against tables with them enabled, Snowflake had V3 in public preview with Variant and row lineage working while default values and geography did not, Trino's connector listed V3 as experimental without column defaults or deletion-vector reads, and Spark picks up V3 features release by release (Mohadata).
  • Small and high-frequency writes still hurt. Metadata overhead and file explosion remain real limitations for small-scale or high-frequency write patterns, the Python, Rust and Go ports lag the Java reference implementation, and write/delete support is incomplete in some popular engines (daily.dev summary).
  • Managed convenience can quietly re-close the stack. Table maintenance and catalog control are exactly where vendor differentiation now lives, and which parts of the stack remain open once a vendor manages them for you is an open question rather than a settled one (Iceberg Lakehouse).

Pros & Cons

Advantages

  • Iceberg has become the de facto interchange layer for the read path across engines, with Delta publishing Iceberg metadata synchronously through UniForm and Paimon registering tables into Iceberg REST catalogs, so a single copy of storage can serve Spark, Trino, Flink, DuckDB, Snowflake and BigQuery workloads.
  • The 1.11.0 release hardens the V3 specification into production defaults — deletion vectors backed by Roaring bitmaps, a native Variant type for semi-structured data, server-side scan planning in the REST catalog, and built-in envelope encryption with KMS support — which removes several of the workarounds that earlier lakehouse teams had to build themselves.
  • Large-scale migrations are now well documented rather than speculative: Grab is moving petabytes across billions of S3 objects off directory-based Hive Parquet onto Iceberg, and open-sourced a UnifiedSparkCatalog that hides table-format differences from its users.

Disadvantages

  • Governance does not travel with the table: the same dataset can be reachable by users, service accounts and AI agents through different engines with inconsistent masking and no centralised policy enforcement, and the Iceberg dev list itself concedes there is no easy way to secure Iceberg data across multiple tools and engines.
  • V3 feature support is uneven across engines — Databricks has deletion vectors generally available while V2 reads break against those tables, Snowflake had V3 in public preview, and Trino's connector listed V3 as experimental — so per-table format-version planning is unavoidable.
  • Iceberg punishes small-scale or high-frequency write patterns with metadata overhead and file explosion, and the non-Java ports in Python, Rust and Go still lag the reference implementation, so maintenance automation and language choice both become production concerns.

Recommendation

Adopt Iceberg as the default table format for shared analytical and AI/ML datasets, but treat the adoption decision as already made and spend your planning effort on operations. The concrete work in this cycle is version and engine matrix management: format-version is a table-level property, so different tables in the same catalog can run different versions, which lets you stage a V3 rollout table by table instead of cluster-wide (Iceberg Lakehouse). Before enabling deletion vectors or Variant, verify each consuming engine on the specific table — deletion vectors can break V2 readers, and V3 support differed materially between Databricks, Snowflake, Trino and Spark (Mohadata).

Scale-out patterns should be borrowed rather than invented. Grab's migration is instructive on two points: abstract the table format behind a catalog layer so producers and consumers are not exposed to per-format differences, and drive the migration from the ingestion and transformation platforms rather than table by table (Grab Engineering). Budget explicitly for maintenance — compaction, snapshot expiry and metadata hygiene — and decide who owns it, whether that is your platform team or a warehouse that offers automatic compaction and lifecycle management for you (AlgeriaTech). Avoid pointing high-frequency, small-batch writers directly at Iceberg tables without a compaction strategy; that is where metadata overhead and file explosion bite (daily.dev summary).

Finally, pair every Iceberg rollout with a centralised access-policy decision point covering human users, service accounts and agents. Open storage plus per-engine policy is the failure mode this cycle: the format will happily let four engines read the same table under four different masking regimes (NHI Mgmt, Apache dev list). Where you are all-in on a single vendor stack, a native format with an Iceberg compatibility layer such as UniForm remains a defensible choice; keep the Iceberg REST catalog as the portability contract you test regularly rather than assume (tech-insider, Sigmoid).

Sources

Overview

Apache Iceberg is an open table format for large analytic datasets that brings SQL-table reliability to data lakes while allowing multiple engines to safely work with the same tables. The Apache project describes Iceberg as a high-performance format for huge analytic tables that supports engines such as Spark, Trino, Flink, Presto, Hive, and Impala working on the same tables concurrently (Apache Iceberg). Its core value is separating table semantics from a single compute engine or warehouse so shared analytical data products can remain open, versioned, and reproducible.

Iceberg's table specification is mature enough for adoption. Versions 1, 2, and 3 of the spec are complete and community-adopted, while version 4 remains under active development and is not yet formally adopted (Apache Iceberg Spec). The spec defines snapshot-based metadata, optimistic commits, serializable isolation goals, schema evolution, partition evolution, position and equality deletes, deletion vectors in v3, and compatibility rules across format versions (Apache Iceberg Spec).

The strongest current adoption driver is open lakehouse interoperability. Iceberg's REST catalog specification defines a common OpenAPI-based API for interacting with any Iceberg catalog, enabling new languages and engines to support catalogs with one client implementation, while supporting secure table sharing through credential vending or remote signing (Apache Iceberg REST Catalog Spec). This makes Iceberg especially relevant for data platforms that need governed access from multiple engines, warehouses, and ML/AI workloads without copying datasets into each platform.

Adoption Signals

  • Apache Iceberg's latest listed release is 1.11.0, and the project states that Iceberg 1.0.0 officially guaranteed API stability after the API had already been integrated with many processing engines (Apache Iceberg Releases).
  • Snowflake supports Iceberg tables across all accounts, cloud platforms, and regions, with read/write support, Snowflake-managed and external catalog options, Snowflake Open Catalog integration, and support for Iceberg spec versions 1, 2, and 3 with caveats (Snowflake Documentation).
  • Databricks announced public preview support for managed Iceberg tables in Unity Catalog, including read and write access from Databricks and external Iceberg engines through Unity Catalog's Iceberg REST Catalog API, plus governance for Iceberg tables managed by foreign catalogs such as AWS Glue, Hive Metastores, and Snowflake Horizon Catalog (Databricks).
  • AWS Prescriptive Guidance describes native Iceberg support across Amazon EMR, AWS Glue, Amazon Athena, and Amazon Redshift for building transactional data lakes on Amazon S3, and states that the next-generation Amazon SageMaker lakehouse is fully compatible with Iceberg and can query data in place using the Iceberg REST API (AWS Prescriptive Guidance).
  • The REST catalog has become a de facto interoperability focus: Snowflake supports remote Iceberg REST catalogs including AWS Glue and Snowflake Open Catalog, while Databricks exposes Unity Catalog through Iceberg REST Catalog APIs for compatible clients such as Spark, Flink, Trino, PyIceberg, Kafka Connect, and Redpanda (Snowflake Documentation, Databricks).

Risks

  • Operational maintenance is mandatory. Iceberg recommends expiring snapshots, removing old metadata files, deleting orphan files, and optionally compacting data files and rewriting manifests; otherwise tables can accumulate metadata, small files, stale snapshots, and unreferenced objects that hurt performance and storage cost (Apache Iceberg Maintenance).
  • Maintenance can be dangerous if misconfigured. Iceberg warns that deleting orphan files with a retention interval shorter than the expected write duration can corrupt a table by deleting in-progress files, and that path-string mismatches on some file systems can lead to data loss during orphan-file removal (Apache Iceberg Maintenance).
  • Catalog strategy is a platform decision, not an implementation detail. Snowflake distinguishes between Snowflake-managed Iceberg tables with full platform support and externally managed Iceberg tables with limited platform support, and notes that Snowflake does not sync remote catalog access control for users or roles in catalog-linked databases (Snowflake Documentation).
  • Engine support is uneven. Snowflake supports some Iceberg v2/v3 features but not equality delete files, has restrictions around external query-engine writes, and documents numerous caveats for external catalogs, row-level deletes, private connectivity, metadata consistency, replication, streams, and fine-grained access control policies (Snowflake Documentation).
  • Open format does not automatically equal open governance. Teams still need a chosen catalog, access-control model, lineage model, data-quality checks, lifecycle policies, cost controls, and ownership conventions across engines; otherwise Iceberg can become another unmanaged data lake layout rather than a governed data-product foundation.

Pros & Cons

Advantages

  • Provides an open, engine-neutral table format for large analytical datasets on object storage.
  • Supports ACID-style table updates, schema evolution, partition evolution, hidden partitioning, snapshots, time travel, rollback, and row-level deletes.
  • Improves interoperability across Spark, Flink, Trino, Presto, Hive, Impala, cloud warehouses, catalogs, and lakehouse platforms.

Disadvantages

  • Requires operational ownership for compaction, snapshot expiration, metadata cleanup, orphan-file removal, and manifest maintenance.
  • Catalog choice can create governance, interoperability, and lock-in trade-offs even when the table format itself is open.
  • Feature support differs across engines, especially around writes, equality deletes, v3 features, external catalogs, and fine-grained access policies.

Recommendation

Adopt Apache Iceberg as the default open table format for shared analytical datasets that need open storage, multi-engine access, reproducibility, governance, and long-lived interoperability. It is especially relevant for AI and ML data foundations because feature pipelines, RAG ingestion, analytics, model evaluation, lineage, and backtesting all depend on consistent, versioned, high-quality data that can be accessed by different engines without duplicating storage.

Adoption should be platform-led rather than project-by-project. Standardize catalog strategy, table naming, ownership, access-control integration, snapshot retention, compaction, orphan-file cleanup, metadata cleanup, branch/tag usage, schema evolution rules, and compatibility expectations across engines. Treat the Iceberg REST catalog as a key architectural boundary, and test real read/write interoperability among the engines that matter before declaring a table "open."

Use managed table services where they reduce operational burden, but keep ownership of the portability contract. Validate which engine or catalog is authoritative for writes, which platform performs maintenance, how access policies are enforced across engines, and what feature subset is safe for production. Move workloads to Iceberg when data products require cross-engine use; avoid adopting it solely as a file-layout change without governance and maintenance automation.

Sources