Overview
Apache Airflow is the open-source standard for scheduling and monitoring batch data workflows as directed acyclic graphs. Airflow 3 delivers a redesigned UI, improved scheduler performance, and continued investment in the Task SDK and provider ecosystem while maintaining backward-compatible DAG concepts (Airflow 3 release).
Adopt for batch ingestion, feature generation, model retraining orchestration, and downstream publishing where you need explicit dependencies, retries, and operational visibility. Use complementary stream engines when latency requirements fall below minute-level scheduling.
Adoption Signals
- Airflow 3 release blog documents migration paths from 2.x with focus on scheduler and API stability.
- Managed Airflow on AWS, GCP, and Astronomer eases upgrades for teams without platform squads.
- Deferrable operators reduce worker slot waste for long external waits common in LLM batch jobs.
- OpenLineage integrations improve cross-system lineage for compliance-oriented data platforms.
Risks
- Upgrade projects from 2.x require testing custom operators and plugin compatibility.
- Running DAGs with embedded LLM calls without cost caps can spike provider bills.
- Shared metadata databases remain single points of failure without HA discipline.
- Overusing PythonOperator for heavy compute starves scheduler throughput.
Pros & Cons
- Airflow 3 modernizes the scheduler and UI while preserving the DAG mental model teams already operate.
- Huge provider ecosystem integrates warehouses, lakes, Kubernetes, and ML training steps in one orchestration plane.
- TaskFlow API and deferrable operators improve ergonomics for Python-native data and AI pipelines.
- Operational complexity remains high: metadata DB, executors, workers, and upgrades need platform ownership.
- DAG sprawl without standards produces untestable pipelines and unclear SLAs.
- Not ideal for sub-second streaming; pair with Flink or stream processors for real-time paths.
Recommendation
Adopt Airflow 3 as the default batch orchestrator for data and ML pipelines, with platform-managed executors and DAG review standards. Plan the 2.x to 3.x migration on a fixed calendar with integration tests on critical DAGs before cutover.
Sources