Apache Airflow 3 Adopt
Overview
Apache Airflow is the open-source standard for scheduling and monitoring batch data workflows as directed acyclic graphs. Airflow 3 delivers a rewritten React UI, DAG versioning, scheduler-managed backfills, event-driven scheduling and a Task SDK that separates task execution from the scheduler, while keeping the DAG model teams already use (Airflow 3 release).
Adopt for batch ingestion, feature generation, model retraining orchestration, and downstream publishing where you need explicit dependencies, retries, and operational visibility. Use complementary stream engines when latency requirements fall below minute-level scheduling.
Adoption Signals
- Airflow 2 reached end of life on 22 April 2026; Airflow 3, first released on 22 April 2025, is the only maintained major line (Airflow version life cycle).
- In Astronomer's State of Airflow 2026 survey (5,800+ respondents, January 2026), 26% of users had already moved to Airflow 3 and 84% of those still on 2.x were planning the upgrade; 32% ran GenAI or MLOps use cases on Airflow in production (Astronomer).
- Amazon MWAA has supported Airflow 3.3.1 since 1 September 2026; the 3.3 line adds a Task and Asset State Store for durable task state and an experimental Language Task SDK for Java and Go tasks (AWS What's New).
- Google Cloud made Airflow 3 generally available in Cloud Composer 3, which it is renaming Managed Service for Apache Airflow, and since 26 August 2026 lets Airflow 3.3.1 and 3.2.2 environments load snapshots from Airflow 2.11.1 for migration (Google Cloud release notes).
Risks
- Airflow 2 no longer receives fixes after its 22 April 2026 end of life, so remaining 2.x environments carry unpatched risk; the upgrade still requires testing custom operators and plugin compatibility. Managed services still offer Airflow 2.11: Google Cloud changed its default Airflow version to 2.11.1 on 28 April 2026, while Airflow 3 builds are available alongside (Google Cloud release notes).
- Running DAGs with embedded LLM calls without cost caps can spike provider bills.
- Shared metadata databases remain single points of failure without HA discipline.
- Overusing PythonOperator for heavy compute starves scheduler throughput.
Pros & Cons
Advantages
- Airflow 3 modernizes the scheduler and UI while preserving the DAG mental model teams already operate.
- Huge provider ecosystem integrates warehouses, lakes, Kubernetes, and ML training steps in one orchestration plane.
- TaskFlow API and deferrable operators improve ergonomics for Python-native data and AI pipelines.
Disadvantages
- Operational complexity remains high: metadata DB, executors, workers, and upgrades need platform ownership.
- DAG sprawl without standards produces untestable pipelines and unclear SLAs.
- Not ideal for sub-second streaming; pair with Flink or stream processors for real-time paths.
Recommendation
Adopt Airflow 3 as the default batch orchestrator for data and ML pipelines, with platform-managed executors and DAG review standards. Airflow 2 is end of life, so treat any remaining 2.x environment as migration debt: move it to Airflow 3 on a fixed calendar, with integration tests on critical DAGs before cutover, and pin new managed environments to an Airflow 3 version explicitly rather than accepting a provider default.
Sources
Consulting
From radar to implementation
You have assessed the technologies — we help you bring them into production systems safely and measurably.
