A six-hour streaming backlog replays in four hours, or in sixty

Someone asks for a real-time dashboard. They mean they’re tired of waiting until tomorrow to find out about today, and that’s fair — a nightly job finishing at 04:00 is a real problem. The fix is almost never a streaming platform.

Short version: the bill for streaming isn’t the processing job. You’re buying a rota, a state store nobody planned to operate, and a replay whose length turns on a ratio most teams never work out.

When is real-time worth paying for?

Real time earns its bill when code acts on the answer and a clock you don’t control decides what counts as late. Get the person who wants it to fill in all four blanks.

______ decides ______ within ______, and if the answer arrives late it costs ______.

If the first blank is a person, add the minutes they need to notice the alert, open the dashboard, work out whether the number is real, then message somebody. Call it four minutes. Four minutes of lag sits inside that reaction time, and cutting it to one second changes nothing about when anybody acts. Hold on to those four minutes; they come back at alert thresholds.

Some answers do justify the bill. A rule decides whether a payment goes through while the customer stands at a card terminal, or a bid has to be in before the auction closes. Code decides, the clock is set outside your control, late is the same as wrong.

Batch, micro-batch and streaming: what each tier costs to run

Running cost rises with the number of systems you agree to keep alive at three in the morning, not with the latency figure in the brief. Freshness below is wall-clock lag.

The bottom row is the trade nobody prices. A transactional sink commits on checkpoint, and a consumer reading with isolation.level=read_committed sees nothing until it does: the Kafka consumer configuration reference says such records “will be withheld until the relevant transaction has been completed”. Flink doesn’t checkpoint until you set an interval, so that floor is one you picked yourself.

Then the money. Snowflake’s warehouse documentation gives the billing unit as “per-second billing (with a 60-second minimum each time the warehouse starts)”, so a load every four minutes is 15 resumes an hour and 15 billed minutes an hour however short the loads are. One source on a high-water-mark column sits at the $2,500 end of the fixed-price band on our data engineering page. Log-based change data capture doesn’t. The processing job is the cheap part.

Watermarks, exactly-once and state: three problems batch never hands you

Streaming hands you three problems a nightly job never does: how long to wait for a late event, whether a duplicate can debit a card twice, and how large the state on disk may grow. In batch you re-run yesterday. In a stream you declare the wait up front, and that declaration is a business decision wearing a config line’s clothing.

CREATE TABLE payments (
  payment_id  STRING,
  amount      DECIMAL(12,2),
  event_time  TIMESTAMP(3),
  -- policy, not a tuning parameter
  WATERMARK FOR event_time AS event_time - INTERVAL '30' SECOND
) WITH ('connector' = 'kafka', ...);

Spark Structured Streaming writes the same line as withWatermark("event_time", "30 seconds"). Thirty seconds, and a phone that sat in a tunnel reports four minutes late. Flink SQL drops those records silently; the only trace is the numLateRecordsDropped counter in the Flink metrics reference, and catching them means dropping to the DataStream API, because allowedLateness and sideOutputLateData live in the windows documentation and nowhere in SQL.

Exactly-once is far cheaper to configure than to understand. Two-phase commit does it for you on a transactional sink — Kafka with DeliveryGuarantee.EXACTLY_ONCE, JDBC over XA — at the price of the checkpoint interval and a transaction.timeout.ms set longer than your worst checkpoint plus your worst restart, not merely longer than the interval. The Flink Kafka connector documentation is blunt: “it is highly recommended to tweak Kafka transaction timeout (see Kafka producer transaction.timeout.ms) >> maximum checkpoint duration + maximum restart duration or data loss may happen when Kafka expires an uncommitted transaction”. The producer default is one minute, and the broker refuses anything above transaction.max.timeout.ms, fifteen minutes by default. Get that wrong and the data is gone.

State is a database on your task nodes that nobody planned to operate. The state backends page gives Flink’s default as HashMapStateBackend, on heap rather than RocksDB on disk, so the first failure is an out-of-memory. What bounds it in the Table API is table.exec.state.ttl, whose default the table configuration page gives as 0 ms: “Default is never clean-up the state.” Nobody changes that until the first disk fills.

Check whether the replay catches up

A replay catches up only when your apply rate beats the live write rate, and the time it takes is the backlog divided by the difference between the two, not by the apply rate. Oxbow is an invented source database behind a parcel-locker network. No system we have run is described here, and every number below is one I chose so that you can recompute all of it.

Oxbow’s change events land in Kafka at a steady write rate of w = 1,200 events a second. A Flink job maintains a thirty-day aggregate from them, and it has been down six hours — 21,600 seconds, so the backlog at restart is 1,200 × 21,600 = 25,920,000 events. The job restores from a savepoint and replays at an apply rate of a = 3,000 a second while new events keep arriving at 1,200, so the net drain is 3,000 − 1,200 = 1,800 a second. Catch-up is 25,920,000 ÷ 1,800 = 14,400 seconds.

Four hours.

A second route gives the same answer: catch-up equals outage × w ÷ (a − w), or 6 × 1,200 ÷ 1,800 = 4 hours. That second form is the whole decision.

Slightly faster than the write rate is not good enough. At 1.1w, six hours down buys two and a half days of replay.

The obvious estimate goes wrong in one place: a isn’t constant. Measure it on an idle cluster and you get the figure from before the state grew; once the working set stops fitting in memory, a falls. Halve it. At a = 1,500 the net drain is 300 a second, catch-up is 25,920,000 ÷ 300 = 86,400 seconds, and a full day still sits inside Kafka’s default retention of log.retention.hours = 168 in the broker configuration reference. Derate a fourfold, to 750, and it never converges at all, because 750 is below 1,200 and the backlog then grows for as long as you allow it.

Retention is the parameter teams treat as fixed, and it’s the one most worth moving. At 1 KB an event, 1,200 a second is 1.2 MB/s; over 86,400 seconds that’s 103,680 MB, call it 104 GB a day, and at replication factor three, 311 GB. Seven days is 2.2 TB, thirty days 9.3 TB, so widening costs about 7.1 TB of disk. One config line buys replay room for every consumer on the cluster, which is when it wins. A compacted topic won’t substitute. Kafka’s design documentation says compaction “retains at least the last known value for each message key within the log of data for a single topic partition”, and the events in between are what an aggregate sums over.

Raising a with parallelism is the other route, and it’s cheaper when keys spread evenly: double the slots, a reaches 6,000, net drain is 4,800, catch-up falls to 5,400 seconds. But parallelism lifts a only as far as the hottest key allows, and if one locker site carries 30% of events no slot count takes you past about 3.3× the single-slot rate. So the runbook number is this: at a = 3,000, the longest outage that still drains inside a day is 24 ÷ 0.667 = 36 hours. Write 36 next to 168 and you have a replay plan. The figure I’d trust least is a, since I’ve seen it read off an idle cluster far more often than a loaded one.

Who holds the pager, and what pages them at 3am

Your rota holds it, not ours. Alerts go to your Slack or PagerDuty, and the runbook is for whoever is on call at seven in the morning. One pager across four engineers is one week in four, each, indefinitely. If nobody wants that week, that’s the finding rather than a detail for later: buy a rota, or build the micro-batch version and pay us less.

A nightly batch that dies at 02:00 can wait until 09:00; a fraud scorer that stops consuming at 02:00 lets payments through unscored for every minute it’s down. Same hour, different alert.

What goes wrong at 3am is rarely a failed job. Consumer lag climbs with nothing logged anywhere, so page on records-lag-max from the Kafka monitoring guide over a threshold agreed in writing and held five consecutive minutes — and those four minutes of human reaction time from the first section are why that threshold gets written in minutes rather than seconds. One malformed message sits at the head of a partition and the job will neither parse it nor skip it, so page on numRecordsOutPerSecond at zero while input is non-zero. Checkpoints time out, almost always because backpressure is holding up barrier alignment rather than because a disk is slow, so alert on lastCheckpointDuration above half the interval. Unaligned checkpoints, from Flink’s checkpointing under backpressure page, trade a bigger checkpoint and a slower recovery for finishing at all. None of it arrives as a red box in an orchestrator.

Adding the second stream costs almost nothing

Adding a second stream to a platform you already run costs a processing job and a few lines of alerting, because the expensive part was bought years ago. The first stream buys the broker, the state store, the schema registry, the rehearsed replay and the week in four. Every stream after it divides that same fixed cost by one more, so the per-stream price falls roughly as 1/n, and the question to settle before anyone quotes is which n you’re standing at. The first stream is expensive. The fourth is not. And if you’re at n = 1 with a four-minute tolerance, the honest recommendation is the one that bills us less: a high-water-mark column, a five-minute schedule, and the pager you already carry.