Apache Fluss Assess

data-platformopen-sourceanalytics-engineeringstreaming-storagestreaming-lakehousereal-time-analyticsapache-flinklakehouse-formatskafka-alternativechange-data-capture

Overview

Apache Fluss is a streaming storage system that positions itself as the real-time data layer for a lakehouse. Rather than treating a log and a table as separate systems, it exposes a common table abstraction over streaming data, continuously updated tables and historical lakehouse data, and integrates with Apache Flink and Apache Spark on the compute side and with Apache Paimon, Apache Iceberg, Apache Hudi and Lance on the storage side (BigDATAwire). It was initiated by the Flink team at Alibaba Cloud in July 2023, open-sourced at Flink Forward Asia 2024, and has now graduated from the Apache Incubator to a Top Level Project with unanimous IPMC approval and ASF Board sign-off.

The reason to look at Fluss is architectural, not incremental. The recurring story in the evidence is teams whose Kafka-plus-wide-table designs became expensive and unstable at scale: Rednote migrated index data from Kafka to Fluss inside a pipeline that feeds search, recommendation and advertising, and Taobao Instant Commerce rebuilt real-time decisioning that previously sat on Kafka, Flink, Paimon and StarRocks. Columnar streaming reads and lakehouse tiering are the levers in both cases.

We place Fluss in assess because the concept is credible and the reference cases are real, but the maturity curve is steep and recent. Zero-downtime schema evolution and consumer-aware snapshot lifecycle management only arrived in 0.9, released March 2026 — precisely the gaps the review board flagged. That is a project moving fast in the right direction, not yet a component we would put under a critical pipeline on the strength of the docs alone.

Adoption Signals

  • Graduated from the Apache Incubator to Top Level Project status, with the graduation proposal receiving unanimous IPMC approval and subsequent ASF Board approval (graduation post).
  • Community metrics at graduation: 157 contributors, 2,000+ GitHub stars and 1,700+ merged pull requests (graduation post).
  • Reported production deployments at Alibaba, Xiaohongshu (Rednote), Fresha, JD.com, Ant Group and iQIYI.
  • A detailed public production case study from Rednote covering columnar streaming, cold-data isolation and lakehouse integration, presented at Flink Forward Asia 2026 (case study).
  • Release cadence is steady and substantive: 0.8 in November 2025 added Flink 2.1 compatibility, Materialized Table support and Helm charts; 0.9 in March 2026 added schema evolution, storage-level aggregations, change data feed, automatic rebalancing and Spark catalog integration.
  • Multi-language access is emerging: Rust, Python and C++ clients 0.1.0 share a single Rust core with Arrow-based data exchange and cover log, primary-key and partitioned tables.

Risks

  • Recently closed capability gaps. Safe schema evolution and consumer-aware snapshot lifecycle management were 0.9 features, so there is little accumulated operational evidence that they hold up under long-running, multi-consumer workloads.
  • Pre-1.0 storage system. The 0.9.1 patch release fixed a Paimon IOManager leak that exhausted /tmp during tiering and correctness bugs in partial update after ADD COLUMN — the kind of defect class you inherit when you adopt young storage.
  • Concentrated vendor and geographic footprint. The project was donated by Alibaba Cloud and its PMC leadership and production references cluster around Alibaba and a handful of large Chinese platforms, which limits independent validation.
  • Immature client ecosystem outside the JVM. The Rust, Python and C++ clients are at 0.1.0; anything beyond Flink or Spark SQL access is early-stage.
  • Architectural lock-in through the table abstraction. Replacing Kafka in a core indexing or feature path is a deep change to data contracts and downstream consumers, and is hard to unwind if the project's trajectory shifts.

Pros & Cons

Advantages

  • Fluss provides a common table abstraction that unifies streaming data, continuously updated tables and historical lakehouse data, which removes a whole class of glue code between a message queue and a table format.
  • It integrates with both Apache Flink and Apache Spark as compute engines and with Apache Paimon, Apache Iceberg, Apache Hudi and Lance as lakehouse formats, so it can slot into an existing open-format stack rather than replacing it.
  • Large-scale production references exist for the exact pain point it targets: Rednote moved a core real-time indexing pipeline off a Kafka-based wide-table architecture that was hitting cost and stability limits, and Taobao Instant Commerce runs real-time decisioning on it.

Disadvantages

  • Capabilities that mature platform teams treat as table stakes arrived only recently — zero-downtime schema evolution and consumer-aware lifecycle management landed in the 0.9 release of March 2026 — so operational experience with them is thin.
  • The project is still pre-1.0, and the 0.9.1 patch release fixed production-relevant defects such as a Paimon IOManager leak causing /tmp disk exhaustion during tiering and out-of-bounds errors on partial update after ADD COLUMN.
  • Nearly all publicly documented production adoption comes from Alibaba and a small set of large Chinese platform companies, and the project was initiated and donated by Alibaba Cloud, so independent operational playbooks and hiring pools are limited.

Recommendation

Run a scoped, time-boxed evaluation rather than a migration. The best-fit candidate is a pipeline that already shows the failure mode Fluss was built for: a Kafka-backed wide table with many downstream consumers each reading a different subset of columns, where cost and stability are the constraint rather than pure throughput. Reproduce that shape on 0.9 or later with your own Flink or Spark jobs and measure read amplification, tiering behaviour into your existing Paimon or Iceberg tables, and recovery time — do not extrapolate from the Rednote or Taobao numbers, whose scale and internal infrastructure differ from most estates.

Stress the two areas the review board called out. Exercise schema evolution against live consumers, including add-column followed by partial updates and deletes, since that path produced fixes as recently as 0.9.1. Then exercise snapshot lifecycle and retention with a lagging consumer present, and confirm you can answer basic operational questions: how you rebalance, how you take a broker out for maintenance, and how tiering behaves when local disk fills. Keep the JVM clients on the critical path for now and treat the Rust, Python and C++ bindings as experimental.

If the evaluation succeeds, hold Fluss in a non-critical or shadow position for at least one more release cycle before promotion, and revisit its ring next quarter against two signals: production write-ups from organisations outside the Alibaba orbit, and a 1.0-class release with stable compatibility guarantees.

Sources