The orchestrator line is five recoveries a month, not fifty pipelines

Five pipelines don’t need Airflow, and fifty can’t do without something. The argument lives in between, and nearly everybody has it in the wrong unit: pipeline count is a proxy for what costs money, and a poor one.

Below about five recoveries a month — pipeline runs that need a human to put them back into a correct state — cron and a shell script are cheaper than Airflow or Dagster. Above it they are not. At five pipelines the line is far off, at fifty it’s behind you, at five hundred the question has changed.

Every operator-minute figure below is chosen, not measured. Swap my inputs for yours.

Is an orchestrator worth it at 5 pipelines?

No. At five daily pipelines an orchestrator costs about 4.6 operator hours a month against cron’s 2.3, and nearly all the gap is its own upkeep.

Five pipelines daily give 5 × 30 = 150 runs a month. I choose a failure rate of one run in sixty, so the recovery load is 150 ÷ 60 = 2.5 runs that need a person. Under cron I choose 50 minutes a recovery: 20 to notice (cron mail gets read at nine), 15 to diagnose, 5 to re-run, 10 to verify. Add 0.8 minutes a pipeline to trace by hand which downstream jobs read stale data: 50 + 4 = 54, and 2.5 × 54 = 135 minutes, 2.25 hours a month, which the table below rounds to 2.3.

The orchestrator leg has two terms. Retries clear a chosen 0.6 of failures with nobody watching, and the 0.4 that reach a person cost 2 minutes to notice, 15 to diagnose, 2 to read the blast radius off the graph, 5 to re-run and 10 to verify. That is 34 minutes, so 0.4 × 34 = 13.6 an average recovery and 2.5 recoveries come to 34 a month — the two 34s are a coincidence, not a typo. Then a chosen 240 minutes, four hours, to own it: 34 + 240 = 274 minutes, 4.57 hours. An orchestrator costs 139 minutes more and schedules the same five jobs.

Worth knowing what “cron” refers to there: POSIX.1-2024 specifies the crontab utility and no daemon; the RATIONALE on the Open Group’s crontab page says why:

All references to a cron daemon and to cron files have been omitted.

What Airflow and Dagster cost to own before they schedule anything

A minimal Airflow installation is five components to own before it schedules anything, and the Architecture Overview page for Airflow 3.3.2 names them: a scheduler, a Dag processor, a Dag bundle, an API Server, and “A metadata database, usually PostgreSQL or MySQL, which stores the state of tasks, Dags and variables.” Four are things to upgrade and the fifth is a database to back up.

My 240 minutes a month is a choice and the second-least certain number here. It splits four ways: 80 for the parse loop and the Dag processor, where the hour lost to a Dag that stopped appearing ends up; 60 for providers and the dependencies they drag in; 60 for the upgrade, one a year at a chosen 12 hours, so 720 ÷ 12 = 60 a month; 40 for the metadata database and its backups. 80 + 60 + 60 + 40 = 240. If yours is two hours, halve every line and the crossover moves with it.

A managed orchestrator just moves that tax rather than removing it. It deletes the 40 for the database and maybe half the 60 for upgrades, because somebody still reads the breaking-changes list and fixes the Dags it broke. 40 + 30 = 70, and 70 ÷ 240 = 29%, so budget about a third off rather than all of it. I’d take that trade at fifty pipelines.

Dagster prices it differently: its unit is the asset, not the task, which matters at five hundred and not at five. Its release line is 1.13, out on 9 April 2026 per the Dagster releases index. Both are free to install. Neither is free to own.

Recoveries a month is the unit, not pipeline count

The number that decides an orchestrator is the count of pipeline runs a month that need a human to put them back into a correct state, and I call it the recovery load.

Solve for where the legs meet. With n pipelines on a daily schedule the recovery load F is 30n ÷ 60 = n/2, cron costs F × (50 + 0.8n) and the orchestrator F × 13.6 + 240. Set them equal: (n/2)(50 + 0.8n) = (n/2)(13.6) + 240, or 0.8n² + 36.4n − 480 = 0. The positive root is (√2,860.96 − 36.4) ÷ 1.6 = (53.49 − 36.4) ÷ 1.6 = 17.09 ÷ 1.6 = 10.7 pipelines, a crossover recovery load of 5.3 a month.

Now price it again with failures left out. Backfills alone, both numbers chosen: one per ten pipelines a month, at 150 operator minutes under cron against 25 with an orchestrator, a saving of 125, so break-even against the fixed term is (n/10) × 125 = 240 and n = 2,400 ÷ 125 = 19.2 pipelines, a recovery load of 9.6 a month.

Two routes, two answers: 5.3 and 9.6. Neither is the line, because each prices one mechanism and ignores the other, so both sit above the true crossover rather than around it.

Start from the tolerance. The failure rate is the number I trust least, so the line has to hold at twice as many failures and at a third as many. Write the rate as one in r and the equation is 0.8n² + 36.4n − 8r = 0. At one in thirty the root is (√2,092.96 − 36.4) ÷ 1.6 = (45.75 − 36.4) ÷ 1.6 = 9.35 ÷ 1.6 = 5.8 pipelines, where F = n, so 5.8 recoveries. At one in 180 it is (√5,932.96 − 36.4) ÷ 1.6 = (77.03 − 36.4) ÷ 1.6 = 40.63 ÷ 1.6 = 25.4 pipelines, and F = 25.4 ÷ 6 = 4.2. Across that 6x swing in my worst number the crossover sits between 4.2 and 5.8, whose midpoint is 5.0.

So the line is five recoveries a month: the middle of that range, and under 5.3, which it had to be. Five recoveries is ten pipelines on a daily schedule and nothing at all on any other schedule, which is rather the point. Five also sits under both crossovers I derived, and the lower line sells more orchestrator builds, so I’d rather say that out loud than have a reader spot it.

One warning about counting against it. A month drawn from a true mean of five carries a standard deviation of √5 ≈ 2.2, so the month that reads 3 and the month that reads 7 can be the same estate.

Operator hours a month at four pipeline counts, on the chosen inputs: one failure in sixty runs, 50 + 0.8n cron minutes a recovery, 13.6 orchestrator minutes and 240 fixed.

At 50 pipelines you are buying retries, backfills and dependency order

At fifty daily pipelines the arithmetic flips hard — 37.5 operator hours a month on cron against 9.7 with an orchestrator — and three mechanisms take nearly all of it.

The numbers first. F = 50/2 = 25 recoveries a month at 50 + 0.8 × 50 = 90 cron minutes each: 25 × 90 = 2,250 minutes, 37.5 hours. The orchestrator leg is 25 × 13.6 + 240 = 580 minutes, 9.7 hours. The saving is 1,670 minutes a month.

Follow one failure through all three. A source extract fails at 02:14 on a Tuesday. Under cron the first signal is mail nobody opens until nine, and the job’s four dependants have already run against yesterday’s data on clock time, with no way to ask whether their input arrived. Under an orchestrator the task retries and clears without waking anyone. Where the retry fails too, the dependants are marked upstream_failed instead of running, so the blast radius is on a screen, not in a head, and the re-run is one backfill instead of five hand-edited date ranges. And what you bought is not the scheduler, since cron actually schedules perfectly well, but the state kept between runs.

Two of those three ship switched on. Retries do not. [core] default_task_retries carries Default: 0 — “The number of retries each task is going to have by default.” Price that. With retries off every recovery reaches a person at the full 34 minutes, so fifty pipelines run 25 × 34 + 240 = 1,090 minutes, 18.2 hours against cron’s 37.5. Retries are worth 25 × (34 − 13.6) = 510 of the 1,670 minutes, 31% of the saving, and losing them moves the crossover from 10.7 pipelines to (√2,800 − 20) ÷ 2 = (52.92 − 20) ÷ 2 = 16.5. A default install pays the whole fixed cost and buys the weakest of the three, which is a real saving and nothing like the one the comparison tables imply.

Double the volume and the gap more than doubles. At 100 pipelines cron is 50 × 130 = 6,500 minutes against the orchestrator’s 50 × 13.6 + 240 = 920, so the saving goes from 1,670 to 5,580, a multiple of 3.3 for a doubling, because cron’s total carries an n² term from the blast radius and the orchestrator’s doesn’t.

Check Airflow’s catchup and max_active_runs defaults before your first backfill

Airflow’s catchup default inverted between version 2 and version 3, and max_active_runs_per_dag has defaulted to 16 through both.

Open two tabs. The Airflow 2.11.0 Configuration Reference gives [scheduler] catchup_by_default as Default: True, and its description opens “Turn off scheduler catchup by setting this to False.” The Airflow 3.3.2 Configuration Reference gives the same option as Default: False and opens “Turn on scheduler catchup by setting this to True.” Same option, opposite instruction, under a minute to confirm. The Upgrading to Airflow 3 guide lists it under Breaking Changes.

The default decides what happens the first time you unpause a Dag with a past start date. The Dag Runs page for 3.3.2 says the scheduler “creates a Dag run only for the latest interval”, while with catchup=True it “will kick off a Dag Run for any data interval that has not been run since the last data interval”. On Airflow 2 that was the default, so a Dag starting six months back replayed six months.

Deliberate backfills use the CLI, and this is the documentation’s own example from that same page:

airflow backfill create --dag-id DAG_ID --from-date START_DATE --to-date END_DATE \
  --reprocess-behavior failed --max-active-runs 3 --run-backwards

The --max-active-runs 3 is the part of their example I’d never drop. Leave the flag off and the ceiling is the Dag’s own max_active_runs, which the CLI reference doesn’t spell out: it gives the flag as “Max active runs for this backfill.” and prints no default under it, where --dry-run two entries above prints “Default: False”. The 16 comes from [core] max_active_runs_per_dag, whose description reads “This is configurable at the DAG level with max_active_runs, which is defaulted as [core] max_active_runs_per_dag”. Sixteen concurrent readers is unremarkable against a warehouse and rude against a transactional database at 02:00, and each run is usually a reload of the kind a full reload that moves data which didn’t change argues against.

How many runs does a Dagster backfill launch?

One per partition. Dagster’s page on backfilling data says, verbatim, “By default, if you launch a backfill that covers N partitions, Dagster will launch N separate runs, one for each partition”, and the sentence after it gives the trade: “However, if you’re using a parallel-processing engine like Spark and Snowflake, you often don’t need Dagster to help with parallelism, so splitting up the backfill into multiple runs just adds extra overhead.” A thousand-partition backfill is a thousand runs until you say otherwise.

Saying otherwise is a code change, not a toggle. The “Launch as…” control “was removed from the UI in Dagster 1.5”, and single-run backfills “are now configured in code using backfill_policy”. The page’s note has two limbs, and the second is easy to lose: single-run backfills “only work if they are launched from the asset graph or asset page, or if the assets are part of an asset job that shares the same backfill policy across all included assets.”

The other direction has a flag that lies, a long way down the reference. Under the heading “Backfill policy” in the Dagster Partitions API reference, the fourth of that page’s six sections, the parameter line for BackfillPolicy.multi_run reads “max_partitions_per_run (Optional[int]) – The maximum number of partitions in each run of the multiple runs. Defaults to 1.” Call BackfillPolicy.multi_run() with no argument and you get one partition per run, exactly what no backfill policy at all gives you. The name says multi and the default says one.

At 500 pipelines the question stops being what failed and becomes what broke

At five hundred pipelines nobody traces failures by hand, because the model above puts the whole cron leg at 1,875 operator hours a month.

Minutes to hours to people, and the last conversion is the one that bites. F = 500/2 = 250 recoveries a month at 50 + 0.8 × 500 = 450 cron minutes each. 250 × 450 = 112,500 minutes. 112,500 ÷ 60 = 1,875 hours. Divided by the 160-hour month this firm uses as its embedded-engineer unit, that is 11.7 people. The orchestrator leg is 250 × 13.6 + 240 = 3,640 minutes, 60.7 hours, 0.38 of a person.

Split the 450 and the regime names itself. Fifty minutes are notice, diagnose, re-run and verify, unchanged since five pipelines. The other 400 are 0.8 × 500, the blast-radius term, which is lineage under a duller name. So 400 ÷ 450 = 89% of a cron recovery at five hundred is somebody working out what reads what, against 40 ÷ 90 = 44% at fifty and 4 ÷ 54 = 7.4% at five. In people: 400 × 250 = 100,000 minutes, 1,666.7 hours, 10.4 of the 11.7, and the other 1.3 do the fixing. A scheduler at five, retries at fifty, and at five hundred a dependency graph nobody ever wrote down. I didn’t see that until the table was on the page.

Nobody pays for 11.7 people, which is where the obvious estimate goes wrong. Cron’s cost never reaches 1,875 hours, because the tracing stops first. The team gives up on which tables read stale data and re-runs everything, so the cost walks off the timesheet onto the overnight window and the warehouse meter, where nobody charges it to orchestration — one of the quieter ways the warehouse bill climbs on flat data.

And none of what you want at this size is a missing cron feature. Run history, dependency order, retry, overlap control: POSIX.1-2024 specifies none of them, and the RATIONALE quoted at the top says why. The daemon was left out, so everything it would have had to remember went out with it. The EXTENDED DESCRIPTION section of the crontab page is one word long.

Both tools answer it by moving the unit of work from the task to the data. Dagster’s Assets page draws the line: “asset definitions know about their dependencies, while ops do not.” Airflow 3 Assets are “a logical grouping of data” whose updates “contribute to scheduling downstream consumer Dags”, though Airflow prints its own caveat: it “makes no assumptions about the content or location of the data”. Your graph is only as true as your discipline in declaring it.

What we would not sell at five pipelines

We would not sell an orchestrator build to a team running five pipelines, and I’d rather lose the work than take it. The thing costs 139 operator minutes a month more than the cron they already have. I’d refuse the other end too: an asset-graph orchestrator bought at fifty pipelines for lineage is five-hundred-pipeline tooling on a fifty-pipeline problem, when retries were the need.

Does the decision flip under the derate? In one unit, not the other. Across that 6x swing the pipeline threshold runs from 5.8 to 25.4, a factor of 4.4, while the recovery-load threshold runs 5.8 to 4.2, a factor of 1.38. Five pipelines stay under at both ends and fifty stay over, so nothing we would tell a team changes. That gap is the whole reason to count recoveries rather than pipelines — a pipeline count is mostly an artefact of a failure rate nobody has measured.

One run in sixty is the input I’m least sure of. The crossover moves roughly as its inverse, and I’ve not metered it over a year on anything I’d publish.

Count your own recoveries for a quarter, not a month, then argue with the inputs; that is most of what our data engineering work is.