In the Apache Kafka 4.4 producer configuration reference, enable.idempotence is marked Importance: low. So are acks, max.in.flight.requests.per.connection, transaction.timeout.ms and transactional.id, the final entry on a page sorted by importance, below ssl.trustmanager.algorithm.
enable.idempotence stops a producer’s own retries from writing duplicates, within one producer session, one topic partition at a time. It survives a network blip. It does not survive your process restarting, and it says nothing about writes that cross more than one partition. Exactly-once needs a transactional.id, a transaction boundary, and a consumer reading read_committed.
What does enable.idempotence actually guarantee in Kafka?
enable.idempotence guarantees that a producer’s own retries don’t write duplicates, inside one producer session and one topic partition at a time, and nothing past that. Its entry in the 4.4 producer configuration reference puts it this way, and the sentence readers stop at is the first:
When set to ‘true’, the producer will ensure that exactly one copy of each message is written in the stream. If ‘false’, producer retries due to broker failures, etc., may write duplicates of the retried message in the stream. Note that enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5 (with message ordering preserved for any allowable value), retries to be greater than 0, and acks must be ‘all’.
Type boolean, default true, importance low. Three conditions arrive as a footnote to the promise of exactly one copy. They are the promise. And the 5 is not a round number somebody liked: the entry for that setting says the broker “only retains at most 5 batches for each producer”, so deduplication runs in a fixed-size window and six in flight means the evidence for the first is gone before the sixth is judged.
Idempotence does not survive a producer restart, and the PID is why
Every producer is handed a fresh producer id (a PID) when it starts, so the broker’s duplicate check begins again at zero the moment your process does. Sequence numbers run from zero upward, one counter per PID and topic partition. The broker accepts a number only when it is one above the last it committed for that pair. A lower number is a duplicate; a higher one is fatal. So the check hangs off producer identity, and identity dies with the process. KIP-98 gives the answer in its Idempotent Producer Guarantees subsection, under the Summary of Guarantees heading: “since each new instance of a producer is assigned a new, unique, PID, we can only guarantee idempotent production within a single producer session.”
Kafka’s own upgrade guide for 3.0 says it too, three-quarters of the way down the page, under its exactly-once semantics heading: “Idempotent delivery ensures that messages are delivered exactly once to a particular topic partition during the lifetime of a single producer.” Lifetime of a single producer. A pod eviction, a rolling deploy, an OOM kill — each ends a lifetime and starts another with a new PID.
And idempotence is per partition. KIP-98’s Motivation section is blunt: “idempotent producers don’t provide guarantees for writes across multiple TopicPartitions.” A loop writing a ledger entry to one topic and an audit record to another runs two unrelated counters. One could land and the other fail.
Kafka 3.0 made enable.idempotence true by default, and 3.0.0 didn’t apply it
KIP-679 changed two producer defaults in release 3.0 — enable.idempotence from false to true, acks from 1 to all — and a bug meant the idempotence half didn’t take effect until 3.0.1, 3.1.1 and 3.2.0. The first bullet of Notable changes in 3.0.0 says what happened: “a bug prevented the idempotence default from being applied which meant that it remained disabled unless the user had explicitly set enable.idempotence to true”. A cluster that went to 3.0.0 and stayed has acks=all and no idempotence, which looks safest in a config file and deduplicates nothing.
The live trap is the other half of that rule, and it’s the first thing I check on a producer I’ve inherited. Set acks=1 and leave enable.idempotence alone and idempotence is turned off without a word: “if conflicting configurations are set and idempotence is not explicitly enabled, idempotence is disabled”. Set acks=1 and enable.idempotence=true and the client throws a ConfigException. Same two settings. Two outcomes, decided by whether somebody typed the word true.
Use transactional.id and isolation.level=read_committed, or you don’t have exactly-once
A transaction boundary is two settings on two different clients, and leaving either at its default leaves you with at-least-once delivery. The Apache Kafka design documentation, under Using Transactions, is explicit: “The consumer configuration must include isolation.level=read_committed and enable.auto.commit=false.” The producer sets transactional.id, which also “makes sure that a restarted application causes any in-flight transaction from the previous instance to abort”.
Three, actually, and the one that gets missed is enable.auto.commit=false, because a consumer on its own commit timer commits offsets outside the transaction.
transactional.id buys what idempotence cannot. Its own entry says it “enables reliability semantics which span multiple producer sessions”, and that without one “the producer is limited to idempotent delivery”. The default is null.
Then the consumer side, where the bill arrives. isolation.level defaults to read_uncommitted. Switch it to read_committed and the consumer reads only as far as the last stable offset, “the one less than the offset of the first open transaction” in the consumer configuration reference, which adds: “read_committed consumers will not be able to read up to the high watermark when there are in flight transactions.” How long one can stay open is transaction.timeout.ms. Nobody sizes it that way.
How to size transaction.timeout.ms: a worked derivation
Size transaction.timeout.ms to outlast the span from the first partition joining the transaction to the commit landing: commit interval, plus commit latency, plus restart. Ferrystone is an invented payments ledger, the one this library reaches for whenever retries or idempotency come up. It is not a client and never was, and every number below is chosen so the arithmetic checks.
Start with units. The producer default is 60,000 ms = 60,000 ÷ 1,000 = 60 s = 60 ÷ 60 = 1 minute, and the broker cap above it is 900,000 ms = 900 s = 15 minutes.
Ferrystone’s loop has four live variables and I’m not adding a fifth. T_c is the commit interval, how often the loop closes a transaction: 30 s. L is the worst commit step, beginTransaction to commitTransaction returning — I’ll accept one commit in a thousand overrunning what I size for, so L is that step’s p99.9, not its mean: 12 s. R is the worst restart, from the process dying to its replacement calling initTransactions(): 180 s. V is the visibility tolerance, the longest a committed entry might sit unseen by a read_committed consumer, and it is derived rather than picked: reconciliation runs every 600 s and needs 120 s of its own, so a stall past V = 600 − 120 = 480 s costs it a cycle.
The clock doesn’t start at beginTransaction. The entry for transaction.timeout.ms: “The start of the transaction is set at the time that the first partition is added to it.” So the floor is T_c + L + R = 30 + 12 + 180 = 222 s.
The obvious estimate goes wrong prudently: the loop commits every 30 seconds, 60 seconds is twice that, the default is already 60,000 ms, so nobody touches it. The hidden term is R. Of the 222 s, the restart is 180 ÷ 222 = 0.81 of the floor. The interval the obvious estimate sized for is 30 ÷ 222 = 0.135, about 13.5%. The Background section of KIP-939 is why R belongs in the sum: “InitProducerId aborts the on-going transaction”, so a prepared one “cannot survive client restart”. What real time costs put that rule qualitatively for a Flink sink: size past the worst checkpoint plus the worst restart, not past the interval. What follows is that sentence with numbers in it.
Headroom next, argued rather than asserted. Each term is a worst case read over a short window, and the two errors cost differently: too low aborts a transaction that would have committed, too high only lengthens a stall. Double it. 2 × 222 = 444 s, rounded up to 450 s = 450,000 ms = 7.5 minutes.
Then the ceiling, from two directions, lower one wins: V = 480 s against transaction.max.timeout.ms = 900 s, and min(480, 900) = 480 s. What has to fit under V is the stall, though, and the stall is the timeout plus the broker’s cleanup sweep: transaction.abort.timed.out.transaction.cleanup.interval.ms defaults to 10,000 ms, so add 10 s. The sweep is a broker constant rather than a fifth Ferrystone variable, so the count holds. And 450 + 10 = 460 ≤ 480, with 20 s spare, so transaction.timeout.ms=450000.
The spare is thinner than it looks. Set the stall equal to the ceiling: 2(T_c + L + R) + 10 = V, so T_c* = (480 − 10)/2 − 12 − 180 = 235 − 192 = 43 s.
Forty-three seconds.
Worst stall against commit interval, at L = 12 s, R = 180 s and a 10 s sweep, in seconds.
| T_c (s) | 2(T_c + L + R) + 10 (s) | Against the 480 s ceiling |
|---|---|---|
| 30 | 454 | 26 s under |
| 43 | 480 | exactly at it |
| 60 | 514 | 34 s over |
The table only checks that algebra. The independent route runs the other way, as a drawdown on the consumer’s 480 s rather than a solve for the commit interval T_c. The sweep comes off the top once, whatever the producer sets: 480 − 10 = 470 s. L and R, the commit step and the restart, are charged twice, because doubling for headroom doubles every term in the floor: 2 × (12 + 180) = 384 s, and 470 − 384 = 86 s. What remains is the commit interval doubled, so 86 ÷ 2 = 43 s again. The drawdown is the one that says why 43 is so small: the restart is paid for twice and the loop gets the change.
Below a 43-second commit interval Ferrystone fits under the ceiling; above it, it doesn’t, and I’d shorten the interval rather than widen the timeout, because widening spends the reconciliation window V came from. Double the interval to 60 s and the answer flips: 514 s, 34 s over. Derate the restart and it breaks twice over. At R = 360 s the required timeout is 2(30 + 12 + 360) = 804 s, which the broker accepts at 96 s under its 900 s cap, but the 814 s stall sits 334 s over the 480 s business ceiling. At R = 720 s the timeout is 2(30 + 12 + 720) = 1,524 s = 1,524,000 ms, 624,000 ms above the cap, and the producer is refused at startup. A three-minute restart survives here. Six minutes breaks the business ceiling, and twelve leaves nothing legal at either end: even the undoubled floor of 762 s overshoots 480 s.
So the rule is a window rather than a number: the transaction window is [T_c + L + R, min(V − sweep, transaction.max.timeout.ms)]. When the floor exceeds the ceiling, no value of transaction.timeout.ms is correct. The commit interval has to come down. But 450,000 ms isn’t free, and the honest figure is 460,000 ms — the worst stall a read_committed consumer sits through behind an abandoned transaction. Half an hour with a calculator puts it in the design rather than in an incident, which is where we put it on data engineering work.
Every Ferrystone term above is a figure I chose. The sweep, the defaults and the cap are Kafka’s. The Metric Changes section of KIP-939, about two-thirds of the way down that page, specifies the broker metric that would make them measured: kafka.coordinator.transaction:type=TransactionStateManager,name=ActiveTransactionOpenTimeMaxMs, “The max time a currently-open transaction has been open”.
Why does the producer fail with InvalidTxnTimeoutException?
Because transaction.timeout.ms is larger than the broker’s transaction.max.timeout.ms, and the rejection lands at initTransactions() rather than at the first send. The producer’s entry says “the request will fail with a InvalidTxnTimeoutException error” — the indefinite article is theirs, not mine. The broker’s entry puts the same refusal from the other side: “If a client’s requested transaction time exceed this, then the broker will return an error in InitProducerIdRequest. This prevents a client from too large of a timeout, which can stall consumers reading from topics included in the transaction.”
Default 900,000 ms, 15 minutes, importance high. The field that bites is Update Mode, which reads read-only: raising the cap is a rolling restart of every broker, not a config you push at two in the morning.
A timed-out transaction isn’t rolled back at its deadline either. The sweeper decides when. transaction.abort.timed.out.transaction.cleanup.interval.ms is “The interval at which to rollback transactions that have timed out”, and it defaults to 10,000 ms, so the rollback waits for the next sweep. That 10 s is the term carried into the ceiling test above, and it is why Ferrystone’s transaction expires at 450 s and might not clear until 460 s.
Apache Kafka 4.1 added an escape hatch with a trap of its own, transaction.two.phase.commit.enable, whose broker half defaults to false. A client that turns its own half on against a cluster that hasn’t is told TRANSACTIONAL_ID_AUTHORIZATION_FAILED — an authorisation failure for a broker setting left at its default.
When exactly-once is not worth turning on
Exactly-once isn’t worth turning on when the sink already upserts on a key: the upsert gives you the end state the boundary was going to buy. The honest case for it, first: one setting applied across every pipeline is cheaper to reason about than a per-sink audit of which writes are idempotent, and that audit goes stale the first time somebody adds a counter column. Kafka’s design documentation says the same from the other end: “In many cases messages have a primary key and so the updates are idempotent (receiving the same message twice just overwrites a record with another copy of itself).” A sink merging on a business key, the shape in the full reload that moves data that didn’t change, lands on the same row either way.
Costed in the units above, the boundary buys up to 460 s of last-stable-offset stall for every read_committed consumer on those partitions: 450 s of timeout plus the 10 s sweep. It also costs enable.auto.commit=false on every consumer, three brokers minimum for the transaction state topic, and a broker cap that can’t move without a rolling restart. Paid for a property the merge already gave you. I wouldn’t turn it on there. It earns all of that where the write isn’t idempotent (a ledger row, a counter increment, an outbound payment call) or where the downstream reads the stream rather than the table.
One admission. ActiveTransactionOpenTimeMaxMs appears nowhere in the Apache Kafka 4.4 monitoring reference. I searched that page on 29 October 2026 and got nothing for it or for TransactionStateManager, and I haven’t checked a running broker’s MBeans. Check before you alert on it. The figure I’d trust least is L, because the producer transaction metrics Kafka documents are cumulative nanosecond totals, so any p99.9 is one you built by hand.
A committed transaction also doesn’t reach the consumer as a unit, the part of KIP-98 that goes unread. Its Transactional Guarantees section says “we cannot guarantee that all the messages of a committed transaction will be consumed all together”. Four reasons follow, none a bug: compaction can overwrite a transaction’s messages, a deleted log segment can take its first part, a consumer can seek into the middle of one, and a consumer needn’t read every partition that took part. Atomicity is a property of the write. Rebuilding it on the read side is your work, not Kafka’s.
Which brings back the configuration page. The five settings that decide whether Ferrystone’s ledger takes the same payment twice all carry Importance: low, and transactional.id is the last entry, below ssl.trustmanager.algorithm. Importance there says how often you touch a setting, not what happens when you get one wrong.