Apache Flink Trial

data-platformmlopsorchestrationstream-processingreal-timeanalyticstrialdatamay-2026apache

Overview

Apache Flink powers stateful stream processing for event-driven analytics, CDC, and real-time features feeding AI systems (Flink).

Trial for AI features that need stateful, exactly-once stream processing with sub-minute freshness, such as real-time features, fraud and anomaly signals, or CDC enrichment. The engine is proven in production; the Trial rating reflects the operating effort, because checkpoint tuning, state backends and Flink SQL need dedicated skills.

Adoption Signals

  • The project lists named production users including Zalando (process monitoring and ETL), Otto Group, Capital One (real-time monitoring and fraud detection), Pinterest (experiment analytics), Lyft (ML feature generation), and Razorpay (AI feature generation and model serving) (Powered By).
  • Active release train: Flink 2.0 (March 2025) came from 165 contributors across 25 improvement proposals (Flink 2.0 announcement), and Flink 2.3.0 followed on 25 June 2026 with changelog SQL operators, materialized-table evolution, and an experimental native S3 filesystem (Flink 2.3 announcement). The GitHub repository has about 26,000 stars (GitHub).
  • AI integration in the engine itself: Flink 2.1 (July 2025) added AI model DDL for Flink SQL and the Table API and extended the ML_PREDICT table-valued function for real-time model inference in SQL queries (Flink 2.1 announcement).
  • Managed services at scale: AWS states that thousands of customers use Amazon Managed Service for Apache Flink (AWS), and Confluent offers serverless Flink on Confluent Cloud (Confluent); IBM completed its acquisition of Confluent on 17 March 2026 (IBM Newsroom).
  • A dedicated sub-project for event-driven agents, Flink Agents, released 0.1.0 in October 2025 (Flink Agents 0.1.0) and 0.3.0 in June 2026, which is still marked as a preview (Flink Agents 0.3.0).

Risks

  • Operational depth. Large-state jobs need tuned checkpoints, alignment, and backpressure handling (Tuning checkpoints and large state). Without people who can debug these, incidents take long to resolve.
  • Breaking 2.x migration. Flink 2.0 removed the DataSet API, the Scala DataStream and DataSet APIs, and SourceFunction, SinkFunction, and SinkV1, so connectors built on them do not work on 2.x; Java 8 support was dropped and Java 17 is the default (Flink 2.0 release notes).
  • Behaviour changes in minor releases. Since 2.3, SQL queries whose upsert key differs from the sink's primary key fail at planning time unless an explicit ON CONFLICT strategy is set (Flink 2.3 announcement). Upgrades need a test run of every SQL job.
  • Preview-grade AI tooling. Flink Agents 0.3.0 is marked as a preview whose APIs and configuration may change in non-backward-compatible ways (Flink Agents 0.3.0).
  • Managed-service lock-in. Managed offerings add their own SQL dialects, connectors, and pricing; the Confluent roadmap now sits inside IBM (IBM Newsroom), so portability of job definitions should be checked.

Pros & Cons

Advantages

  • A mature Apache project with a long production record: Zalando, Otto Group, Capital One, Pinterest, and Uber run it, and Lyft and Razorpay use it to compute machine-learning features in real time.
  • Flink 2.x targets cloud and AI workloads directly, with disaggregated state storage on distributed file systems (2.0), AI model DDL and the ML_PREDICT function for calling AI models from Flink SQL (extended in 2.1), and materialized tables that can evolve without re-ingesting historical data (2.3).
  • Several managed options reduce operating effort, including Amazon Managed Service for Apache Flink and Confluent Cloud for Apache Flink, next to self-hosting with the Kubernetes operator.

Disadvantages

  • Operating stateful streaming is hard: checkpoint sizing, backpressure, state growth, and upgrades need dedicated skills, and a batch lakehouse pipeline is simpler when minute-level freshness is not required.
  • Flink 2.0 removed the DataSet API, the Scala APIs, and the legacy SourceFunction and SinkFunction interfaces, so 1.x jobs and connectors built on them need rework before upgrading.
  • The agent-oriented pieces are early: Flink Agents is still a preview (0.3 in June 2026) whose APIs may change incompatibly.

Recommendation

Flink is rated Trial on this radar. The engine is proven at scale, and the broader pattern of real-time streaming for AI context counts as a safe default on this radar, but running stateful jobs well takes skills that most mid-market teams still have to build. Use it on real projects where AI features, fraud or anomaly signals, or agent triggers must be fresh within seconds, where CDC streams need joins and aggregations before they reach the lakehouse, or where results must be exactly-once. If hourly or daily freshness is enough, batch or micro-batch pipelines on the existing lakehouse are cheaper to run and staff.

Prefer a managed service such as Amazon Managed Service for Apache Flink or Confluent Cloud for Apache Flink over a self-run cluster, and use the Kubernetes operator only where the platform team already runs Kubernetes in production. Start on Flink 2.x with Flink SQL rather than porting 1.x DataStream code. Build these skills before the first job goes live: checkpoint and savepoint handling, choosing and sizing the state backend, event time and watermarks, diagnosing backpressure, and Flink SQL including the changelog and upsert-key semantics that 2.3 made explicit. Set up checkpoint storage, state-size monitoring and a tested savepoint-based upgrade path. Call models through ML_PREDICT asynchronously with timeouts and cost budgets. Flink Agents is an early-stage preview; keep agent logic that must be stable outside its APIs until they reach 1.0.

Sources

Consulting

From radar to implementation

You have assessed the technologies — we help you bring them into production systems safely and measurably.

AI agents

AI agents

Agents in production, not just in a prototype: RUBINLAKE AI Agents handle research, forecasting and market monitoring for your team. Source-backed and operated by us.

Learn more
AI Trainings

AI Trainings

We make your teams AI-ready. Our trainings turn abstract buzzwords into applicable skills. Clear. Practical. Built for doers.

Learn more