Monolithic Jupyter Pipelines Hold

Overview

Monolithic Jupyter notebooks that encode entire ETL, training, and deployment pipelines create untestable, unschedulable assets with hidden state and poor collaboration (Jupyter).

Hold for new production pipelines. Migrate to modular packages, orchestrators (Airflow, Dagster), and versioned datasets with notebooks limited to exploration.

Adoption Signals

• A large-scale study presented at MSR 2019 analysed 1.16 million notebooks from 264,023 GitHub repositories: of 863,878 attempted executions only 24.11% ran without errors and 4.03% reproduced the stored results, and 36.36% of notebooks with an unambiguous execution order had cells run out of order (Pimentel et al., MSR 2019).
• A later comparison of notebooks and Python scripts found that notebooks have lower complexity but 1.4 times more stylistic issues, with code that reads as more entangled (Grotov et al., 2022).
• Platforms that allow scheduling notebooks steer toward version control: Databricks recommends a remote Git repository for versioning assets scheduled with jobs (Databricks notebook task).
• Orchestrators treat a notebook as one step in a larger graph rather than as the pipeline: Dagstermill passes inputs and outputs between notebooks and other assets and aggregates their logs (Dagster Jupyter integration).
• Newer notebook tools address the root causes: marimo stores notebooks as pure Python, re-runs dependent cells reactively and makes notebooks executable as scripts (marimo docs).

Risks

• Hidden state. Cells executed out of order or edited after running leave the kernel in a state no one can reconstruct; the MSR 2019 study names hidden state and execution order among the main causes of failed reproduction (Pimentel et al., MSR 2019).
• No unit of retry or backfill. When extraction, feature engineering, training and publishing share one kernel, a failure late in the run means re-running everything, and partitions or single steps cannot be backfilled.
• Untested production logic. Functions defined in cells are hard to import into test suites, so business rules and data transformations reach production without automated checks.
• Leaky artefacts. Outputs are stored inside the .ipynb JSON, so sample rows, personal data or credentials printed during a run end up in Git history and code review tools.
• Weak audit trail. Without versioned code, parameters and input datasets per run, teams cannot show which data and logic produced a model, which regulated environments and model risk reviews require.

Recommendation

Hold: do not start new production pipelines as single notebooks. Keep notebooks for exploration, prototyping and reports, and move reusable logic into versioned Python packages with unit tests. Run pipelines as separate steps in an orchestrator such as Airflow, Dagster or Prefect, so each step can be retried, backfilled and monitored, and version datasets and model artefacts per run.

For existing notebook pipelines, start with the most business-critical one: extract functions into a package, add tests around the transformations, strip outputs before commit, and let the orchestrator call the package. Where a notebook must stay in the loop, for example a parameterized report, run it through Papermill or Dagstermill as one explicit step with fixed parameters rather than as the whole pipeline. Teams that want notebook ergonomics without hidden state can evaluate marimo, which stores notebooks as plain Python.

Sources

• JupyterLab
• nbconvert
• Papermill
• Pimentel et al.: A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks (MSR 2019, preprint)
• MSR 2019 paper page
• Grotov et al.: A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts (arXiv)
• Databricks notebook task for jobs
• Dagster Jupyter integration (Dagstermill)
• marimo docs