Ingestion and change data capture, lakehouse and warehouse builds, streaming scoped to the decision it changes, modelling, governance and cost. Written from the arguments we have had with ourselves.
What Airflow or Dagster is worth at 5, 50 and 500 pipelines, costed in operator minutes, and the threshold below which we won’t sell a build.
A calculator for warehouse credits run on numbers you can state: credits per query-run, the idle tail, and the 600-second default that sets the break-even.
What enable.idempotence guarantees in Kafka, what a transaction boundary adds, and how to derive transaction.timeout.ms rather than guess at it.
Why full reloads get slow, how log-based change data capture fixes it, what has to be in place in the source database before it is safe, and when reloading the lot is still the right answer.
The replication slot is what breaks when you stream Postgres to a lakehouse: a stopped consumer pins write-ahead log on the source until the volume fills.
Two mechanisms get blamed for the Snowflake auto-suspend setting and only one of them is real. The crossover, derived from the gap between queries.
Spark prices every file open at 4 MB of scanning, which puts the small-file crossover at 4 MB of mean size, about 40x before listing becomes the bottleneck.
Iceberg and Delta both commit by swapping one metadata pointer. The catalog holds that pointer, so the catalog decides which engines can commit.
A worked replay calculation for a streaming pipeline, the latency floor exactly-once imposes, and why the first stream costs a rota rather than a processing job.
A dbt merge scans the target and rewrites whole files, so the crossover isn’t a ratio in rows: the rebuild budget puts it at 0.00053% churn.
Where the money goes when a Snowflake or Databricks bill rises on flat data: count the meters, attribute per query, check auto-suspend, then warehouse size.
Browse by tag
Twenty minutes, no deck. We’ll tell you straight whether this is work we should be doing, and if it isn’t, we usually know who should.
We come back within one business day.
Analytics
We’d like to use Google Analytics to see which pages people find useful. It sets cookies, and only if you say yes. Your choice can be changed at any time under Cookie settings. Privacy