Both formats commit by swapping one pointer, and neither of them holds it. The pointer lives in a catalog, a separate piece of software from the table format. You buy it separately, and somebody else often owns it. Spend a week comparing Parquet layouts and you’ll have decided nothing.
Iceberg and Delta commit the same way. A writer stages immutable files, then asks one authority to swap a single pointer from the old table metadata to the new one. That authority is the catalog. Which engines can reach it, and what it does when two of them swap at once, is the whole decision.
(Keep one error type in your head while you read: CommitStateUnknownException. You’ll meet it again in the last section, and it’s the reason a catalog is an operational choice rather than a procurement one.)
Iceberg or Delta: what the file layouts no longer decide
The file layouts no longer decide read performance, row-level deletes or schema evolution. Iceberg and Delta converged on the same answers to all three. Both are read over Parquet in every deployment worth arguing about — though only Delta’s spec requires it, and Iceberg’s version 1 covers ORC and Avro too. Both now encode row-level deletes as binary deletion vectors instead of rewriting whole files.
Real differences remain, and they aren’t the ones argued about in procurement meetings. The Apache Iceberg table spec, under Format Versioning, says versions 1 to 3 “are complete and adopted by the community” while version 4 “is under active development and has not been formally adopted”. Version 4 restructures manifest metadata into a root manifest and bars new equality delete files, which Delta has no equivalent of; and Delta’s Iceberg-compatibility features both need deletion vectors to be inactive.
But for most teams none of that settles anything. What settles something is the question the data-layout sections don’t answer: who’s allowed to change the table, and what happens to the second engine that tries.
How an atomic swap of the table metadata pointer works
An atomic swap of the table metadata pointer is a check-and-put: the writer says which version it read and where the new metadata sits, and the authority accepts only if the version it holds is still that one. The Apache Iceberg table spec puts it in one line under Optimistic Concurrency.
An atomic swap of one table metadata file for another provides the basis for serializable isolation.
Readers aren’t affected until they refresh, because the same section says they “use the snapshot that was current when they load the table metadata”. Neither side blocks the other, and that’s free in both formats.
Where the swap happens is not free. The spec’s Metastore Tables section puts the pointer “in a metastore or database that is updated with a check-and-put operation”. Its immediate predecessor does the same swap by renaming <random-uuid>.metadata.json to v<V+1>.metadata.json — and carries the sentence that makes reading past the first screen worth the trouble. The spec names rather than numbers its sections, so I’ll cite the heading instead of inventing a number: File System Tables, two thirds down a page whose Overview is what everyone quotes.
Note: This file system based scheme to commit a metadata file is deprecated and will be removed in version 4 of this spec. The scheme is unsafe in object stores and local file systems.
Delta’s filesystem commit has the same shape, with a version number as the pointer. The Delta Transaction Log Protocol, under Delta Log Entries, says Delta files “are the unit of atomicity for a table, and are named using the next available version number, zero-padded to 20 digits”. Whoever creates _delta_log/00000000000000000042.json first owns version 42. What makes that creation exclusive is required but unspecified, which surprised me. Under Creation of New Log Entries the protocol says writers “MUST never overwrite an existing log entry” and should “use atomic primitives of the underlying filesystem”, then stops there. Which primitive is the LogStore’s problem. On S3 there isn’t one: Delta Lake’s storage configuration documentation says S3 “does not provide mutual exclusion, that is, there is no way to ensure that only one writer is able to create a file”, so the default mode wants every concurrent write to come from one Spark driver, and the multi-cluster LogStore “uses DynamoDB to provide the mutual exclusion that S3 is lacking”.
What happens when two engines commit to the same table at once?
One of them loses, and the loser finds out from an error whose type says whether a retry is safe. The Apache Iceberg table spec is blunt under Commit Conflict Resolution and Retry: “When two commits happen at the same time and are based on the same version, only one commit will succeed.” Whether the loser can retry turns on what it asserted. An append asserts nothing about which files are present, so a losing appender replays onto whatever version won. A replace, or a delete that named specific files, has to check those files are still in the table before it goes again. A delete written as a predicate is the exception: the spec says predicate deletes “can always be applied (e.g., where timestamp < X)”, because there is no file list to invalidate.
In the REST catalog that assertion is literal. A commit is a POST carrying two arrays, and the first is the conflict detector. The shape, not a body that would validate:
json
{
"requirements": [{ "type": "assert-ref-snapshot-id", "ref": "main", "snapshot-id": "<the id you read>" }],
"updates": [
{ "action": "add-snapshot", "snapshot": { "snapshot-id": "<the id you wrote>" } },
{ "action": "set-snapshot-ref", "ref-name": "main", "type": "branch", "snapshot-id": "<the id you wrote>" }
]
}The Apache Iceberg REST Catalog OpenAPI document calls the two halves requirements and updates, and requires a server to “fail with a 400 status code if any unknown updates or requirements are received”. Lose the race and the same document hands you a 409: “Conflict - CommitFailedException, one or more requirements failed. The client may retry.”
Delta catches the same collision from the other end, by tracking which files a transaction read rather than asserting a snapshot id. The Delta Lake concurrency-control documentation publishes the matrix. INSERT against INSERT cannot conflict. Compaction against INSERT cannot conflict. Anything involving UPDATE, DELETE or MERGE INTO can, and the failure arrives as ConcurrentAppendException, ConcurrentDeleteReadException, ConcurrentDeleteDeleteException, MetadataChangedException or ProtocolChangedException. Different names, one mechanism: read a version, assert it, lose.
Delta catalog-managed tables moved the source of truth in 2025
Delta moved the source of truth into the catalog in June 2025, not this year. Delta Lake 4.0.0, published 9 June 2025, shipped preview support for catalog-managed tables under a feature then called catalogOwned-preview, and the Delta Transaction Log Protocol states the consequence in its opening lines: “With this feature enabled, the catalog that manages the table becomes the source of truth for whether a given commit attempt succeeded.” Filesystem access to such a table is unsupported on purpose: a writer that skips the catalog can corrupt both.
A commit then takes one of three shapes. The client writes a staged commit to _delta_log/_staged_commits/<v>.<uuid>.json and asks the catalog to ratify it; or sends it inline for the catalog to store as it likes; or, where PUT-if-absent is available, the catalog ratifies and publishes in one step by writing _delta_log/<v>.json itself. Ordering isn’t negotiable. A catalog “must not ratify version v until it has ratified version v - 1, and it must ratify version v at most once”.
Now the quirk, and it’s the kind that eats an afternoon. Delta renamed the feature from catalogOwned-preview to catalogManaged in 4.0.1, published 16 January 2026, and put it at the top of that release as a “[Spark] Breaking change”. In a patch release. The table-ID property moved with it, ucTableId becoming io.unitycatalog.tableId, and the Delta Lake 4.0.1 release page says only that calls sending the legacy key “are handled for compatibility during creation”. Under a minute to confirm.
I wouldn’t turn catalogManaged on for a table that only one engine writes, and the protocol says why. Its appendix of valid feature names classifies catalogManaged as “Readers and writers”, so every reader that hasn’t implemented the feature loses the table outright. Maintenance goes too. Under Maintenance Operations on Catalog-managed Tables, again well past the first screen, the protocol says “By default, maintenance operations are prohibited unless the managing catalog explicitly permits the client to run them”. Checkpoints, log compaction and version checksum are the only exceptions; data files cleanup, for example VACUUM, is not. The restriction has widened since — Delta Lake 4.2.0 of 16 April 2026 blocks OPTIMIZE and REORG run by filesystem path against a catalog-managed table, “consistent with the existing VACUUM restriction introduced in Delta 4.1”. And the status hasn’t moved. Delta Lake 4.1.0 says of the Spark support that “This is still in preview and production usage is not recommended”; 4.2.0 still calls the Unity Catalog managed-table work experimental, and 4.4.1 of 5 October 2026 was tightening maintenance blocks rather than declaring anything generally available.
Delta UniForm lets Iceberg clients read, not write
Delta UniForm lets Iceberg clients read a Delta table and nothing more. The Delta Lake documentation on Universal Format is four words long on the point: “Writes are not supported.” The same page warns that an external write can cause data loss, because an Iceberg writer might run cleanup Delta knows nothing about. So I don’t sell UniForm as interoperability. It’s a read replica in a second dialect — a useful thing to be.
UniForm’s cost is staleness, and the obvious estimate of that staleness is wrong. The same page says why:
Delta Lake ensures that only one metadata generation process per format is in progress at any time in a single cluster. Commits that would trigger a second concurrent metadata generation process successfully commit to Delta, but do not trigger asynchronous metadata generation.
One boundary before the arithmetic. That mechanism is the asynchronous post-commit path, which the same page pairs with a requirement that “Hive Metastore (HMS) must be configured as the catalog”. Delta Lake 4.2.0 of 16 April 2026 moves Iceberg metadata generation “directly into the commit transaction rather than a post-commit hook” for Unity Catalog and starts dropping HMS, which “does not support catalog-managed tables”. Where a release and the docs page disagree I go with the release, so what follows describes the HMS path.
Pick two numbers, both chosen rather than measured by us: a Delta commit every 60 seconds, and a metadata generation taking 90 seconds. The obvious estimate is that an Iceberg reader trails by the generation time, 90 seconds. Wrong, because a commit landing mid-generation gets no generation of its own. The next one starts at the first commit after the last finished, so the cycle is 90 rounded up to a commit boundary: 90 / 60 = 1.5, rounded up to 2, so 2 x 60 = 120 seconds. Fresh Iceberg metadata is already 90 seconds old; just before the next publication it’s 90 + 120 = 210. Mean staleness is (90 + 210) / 2 = 150 seconds, or 150 / 60 = 2.5 minutes.
Check it a second way, from the status property UniForm publishes. Poll the table and converted_delta_version sits some versions behind current; take that gap as 2. Two versions at 60 seconds each is 2 x 60 = 120 seconds of unconverted commits, and the newest commit is itself on average half an interval old, 60 / 2 = 30 seconds. 120 + 30 = 150 seconds again, from a number you read off the table.
Set the bar before judging that number. We pick five minutes for “near real time” and defend it when somebody asks for the phrase without a figure. It’s a number we chose, not one we measured. Against 90 to 210 seconds with a mean of 150, UniForm clears five minutes easily and fails a 60-second bar everywhere in the cycle, because even the floor is 90. Derate the generation threefold, to 30 seconds: the cycle collapses to one commit, floor 30, ceiling 30 + 60 = 90, mean 60. A 60-second bar goes from impossible to borderline, and I still wouldn’t promise it.
Now double the commit rate, to one every 30 seconds against the same 90-second generation. The cycle becomes 90 / 30 = 3 commits, or 90 seconds, so mean staleness is 90 + 90 / 2 = 135 seconds — slightly fresher. The version gap moves the other way: it runs 3, 4, 5 across the cycle, a mean of 4 against the earlier 2. Alert on converted_delta_version lag in versions, as the obvious monitor does, and doubling the commit rate doubles your alerts. The data got fresher. Cadence decides a streaming bill too, which what real time costs works through, and the generation runs on the write’s own compute, so it lands on the meter in why a warehouse bill climbs on flat data.
Check which engines must commit, not which engines read
Count the engines that have to commit to the table, because that count decides whether the format choice costs you anything. Readers are solved: Iceberg and Delta both have engines, and Delta UniForm covers the Iceberg reader against a Delta table. Writers aren’t solved. So here’s the threshold we defend across this library, and we call it the second writing engine.
A shop is above the line when two or more engines must issue commits against one table and at least one can’t authenticate to the other’s catalog. Below the line (one committing engine, any number of readers), pick whichever format your platform defaults to and move on. If exactly one engine ever commits to the table, the choice is close to irrelevant and I wouldn’t spend more than an afternoon on it.
An admission while we’re here. I had Snowflake filed as read-only against externally managed Iceberg tables, and that stopped being true. Snowflake’s documentation on Apache Iceberg tables now says it “supports writes to externally managed tables that use a remote Iceberg REST catalog” for table spec version 2 or 3, with an exclusion list that includes multi-statement transactions and equality delete files. Checked 29 October 2026.
When the answer flips, and what I’d spend the week on instead
The answer flips when a second engine has to commit and can’t commit through your catalog. Databricks publishes the cleanest proof. The table below reproduces Databricks’ own support matrix for what external Apache Iceberg clients can do through the Unity Catalog Iceberg REST endpoint; the unit is support, not speed.
| Table in Unity Catalog | Iceberg read | Iceberg write |
|---|---|---|
| Managed Iceberg | Yes | Yes |
| Foreign Iceberg | Yes | No |
| Managed Delta, Iceberg reads enabled | Yes | No |
| External Delta, Iceberg reads enabled | Yes | No |
Same catalog, same storage, same Parquet files underneath, per Databricks on access from Apache Iceberg clients. Format and management mode decide together whether the second writer gets in: only a Unity Catalog managed Iceberg table takes an external commit, which is why UniForm can’t carry you over that line. And the catalog keeps deciding afterwards. Unity Catalog “blocks external engines from committing duplicate data files to the table”, because the Iceberg spec forbids duplicates inside one snapshot.
So, CommitStateUnknownException. On a commit the Apache Iceberg REST Catalog OpenAPI document describes 500 as “An unknown server-side problem occurred; the commit state is unknown”, and gives 502 and 504 their own wording for the same predicament. All three carry CommitStateUnknownException as the error type in the example body.
Your commit might have landed.
No file format removes that: it’s a property of committing over a network, and the best argument for spending the week on the catalog instead of the format. That week goes on credential vending, the commit retry path, and a handler for the unknown-state case that reconciles rather than retries blind. If you’d rather not spend it, that’s the shape of work our data engineering practice does.
Two soft spots before you go. I can never hold the UniForm version boundaries in my head: Delta 3.1 and above to write a UniForm Iceberg table, 3.2 for Hudi, 3.3 before ALTER TABLE ... SET TBLPROPERTIES will enable it on an existing table. I look all three up every time. And the interop contract everyone is standardising on is thinner than its adoption suggests — the Apache Iceberg REST Catalog OpenAPI document still declares version: 0.0.1 on line 27 at the apache-iceberg-1.12.0 tag of 30 September 2026, while AWS Glue’s Iceberg REST extension endpoint ships APIs “which are not present in the Apache Iceberg REST specification”.