Rotating the secret removes none of the six copies

Six places, and a rotation reaches none of them. A client secret handed to a Spark job is written out by six separate mechanisms, two inside Spark’s reach and four in systems Spark has never heard of. Rotation changes what the identity provider accepts and nothing whatever about where the old string sits. That gap is where most “but we rotated it” reviews go wrong.

Why do rotated secrets keep working?

A rotated secret keeps working because rotation deletes no copy of it. Rotation ends the interval in which a copy is worth anything, and that is the whole of what it does.

Replacing that value reaches nothing else. Not the submit script, not git history, not the event log, not the scheduler’s run records, not the driver’s stdout. Every copy survives the operation, and what ends is the interval in which one still opens the door.

So the useful question after a leak isn’t where the string went but how long it stayed live, which is arithmetic rather than forensics. Microsoft has already published the position this article argues from. Add and manage app credentials in Microsoft Entra ID says it outright: “Client secrets are less secure than certificate or federated credentials and therefore should not be used in production environments.” The same tab caps a secret at “two years (24 months) or less”.

Where a client secret in a Spark config actually ends up

A client secret handed to a Spark job ends up in six places, and only two sit anywhere Spark can reach. Kettlebridge is a fiction I’ll carry through this article: thirty-one people, mooring and berth management software, a market this firm has never sold into. Its numbers are picked so the division can be checked, not read off anybody’s cluster. Its job pulls berth telemetry from ADLS Gen2 every two hours, and the client secret sits in the submit line as --conf spark.hadoop.fs.azure.account.oauth2.client.secret=….

Putting it there isn’t laziness, it’s precedence. The Apache Spark configuration documentation sets the order out under dynamically loading Spark properties: “Properties set directly on the SparkConf take the highest precedence, then those through –conf flags or –properties-file passed to spark-submit or spark-shell, then options in the spark-defaults.conf file.” A flag on the submit line beats the defaults file on every node. The job runs first time, and wiring up a secret store does not. (The same line carries a --jars entry, which has a problem of its own. Hold that thought.)

Now count. One: the submit script in the working copy. Two: git history, in every clone and every fork. Three: the scheduler’s job definition, plus a run record per execution, which at a two-hour cadence is 30 × 24 / 2 = 360 records in a thirty-day window. Four: the driver process, where the string lands three times over — in sun.java.command, in the SparkConf under the key as written, and in the Hadoop Configuration as fs.azure.account.oauth2.client.secret, since SparkHadoopUtil.appendSparkHadoopConfigs copies spark.hadoop.-prefixed properties across with the prefix stripped. Five: the event log file, which EventLoggingListener writes from the SparkListenerEnvironmentUpdate that SparkEnv.environmentDetails builds out of those three. Six: driver stdout and stderr, if the job printed its configuration.

Five of the six outlive the application. The driver is the exception and takes its copy with it. The event log stays, and the history server re-serves that same file at /applications/[app-id]/environment, which Spark’s monitoring documentation lists as “Environment details of the given application.”

Rotation removes zero of the six.

What spark.redaction.regex redacts by default, and what it misses

The default of spark.redaction.regex is (?i)secret|password|token|access[.]?key, and Spark runs it as a substring search over the property key first and the value second, not as a whole-key match. Four alternations. That deny list decides which of the six places carry a redaction marker and which get written out in clear.

Finding the rule is harder than reading it. The row sits in the Runtime Environment table of the Apache Spark configuration page, between spark.executorEnv.[EnvironmentVariableName] and spark.redaction.string.regex, about 13% down the rendered text:

curl -s https://spark.apache.org/docs/latest/configuration.html | python3 -c '
import sys, re, html
t = re.sub(r"(?is)<(script|style).*?</\1>", " ", sys.stdin.read())
t = re.sub(r"\s+", " ", html.unescape(re.sub(r"(?s)<[^>]+>", " ", t)))
i = t.find("spark.redaction.regex")
print(i, len(t), round(100 * i / len(t), 2))'

Against the Spark 4.2.0 documentation on 29 October 2026 that printed 27846 211390 13.17. Those absolutes drift with the page; the ratio doesn’t.

“When this regex matches a property key or value, the value is redacted from the environment UI and various logs like YARN and event logs.” That is the entire specification, and it has sat in the table cell since version 2.1.2.

Places four and five sit inside that reach, because every key the secret lands under carries the substring secret. One, two, three and six do not; a regex in a driver JVM has no opinion about a script on a laptop or a git object in a fork. And “various logs like YARN and event logs” is not an enumerable set, so I’d check what my cluster manager writes rather than trust the word “various”. I don’t know the full list.

Now the jar from earlier. EventLoggingListener exempts one category by name — val noRedactProps = Seq("Classpath Entries"), line 317 of EventLoggingListener.scala on tag v4.0.0 — and SparkEnv files every --jars entry under it. So a jar URI carrying a SAS token reaches the event log in clear whatever the regex says. Open the file, search noRedactProps, confirm it inside a minute.

Default of spark.redaction.regex by Apache Spark release tag, read from internal/config/package.scala, 29 October 2026.

Four tags shipped with access keys off the list altogether, across the 3.0 and 3.1 lines. The optional ? that matches accessKey arrived in 3.4.4, 3.5.2 and 4.0.0, in commit 1e17c392, [SPARK-48930][CORE] Redact awsAccessKeyId by including accesskey pattern, and backported, which is why one 3.4 tag sits in a different row from the three before it.

Widening the pattern is not worth a sprint, and I wouldn’t bill anyone for one. It is a deny list over names somebody guesses in advance, and the best version still reaches two places out of six. Spark’s own comment in Utils.scala points the other way: it searches values as well as keys because sun.java.command carries the whole command line, and its remedy when that over-fires is that “user would have to make the spark.redaction.regex property more specific”. Narrower, not wider. Read a redaction marker as a tripwire rather than a control, and spend the sprint on the credential, the trade EPSS makes against a severity score.

How long does a leaked secret stay valid after rotation?

For the whole of the rotation period, because the copy gets made at the moment of rotation rather than at a random point inside it.

Start from a tolerance, not a number somebody liked. I would not accept a credential that still works a month after it leaked, so T = 30 days and R falls out of it. Take the leak as independent of the schedule: the paste lands at a uniformly random t inside a period of length R, so remaining validity is R − t, uniform on (0, R]. Its 90th percentile is 0.9R, so 0.9R = 30, R = 30 / 0.9 = 33.3, call it 33 days, mean exposure R / 2 = 16.5 days.

Nobody pastes a client secret at a random moment, though, and that timing is the hidden factor that breaks the estimate above. They paste it when the new value exists and the old one has stopped working: the instant of rotation, every period. Remaining validity is R, not R − t. Its 90th percentile is R, so the period collapses to R = T = 30 days. The naive model understates it by exactly two: 33 days of true exposure against a mean of 16.5, and 33 / 16.5 = 2. The correction always costs that factor, because it replaces a mean of R/2 with the whole of R.

Now the units, one rung at a time. 30 days is 30 × 24 = 720 hours, and 720 × 60 = 43,200 minutes. Microsoft’s Access tokens in the Microsoft identity platform page supplies the divisor: “the Microsoft identity platform assigns a random value ranging between 60-90 minutes (75 minutes on average) as the default lifetime of an access token.” Divide: 43,200 / 75 = 576.

A second route reaches that figure without dividing the window by anything. A workload holding continuous access takes a fresh token every 75 minutes: 60 / 75 = 0.8 tokens an hour, 0.8 × 24 = 19.2 a day, 19.2 × 30 = 576 across the window. The routes differ in what you can falsify. Roughly nineteen token requests a day for one service principal is something the Entra sign-in logs will tell you by Thursday. 576 is not.

Take the token lifetime to the 10-minute floor that configurable token lifetimes permits and both routes move together: 43,200 / 10 = 4,320 from the top, 1,440 / 10 = 144 tokens a day times 30 = 4,320 from the bottom.

Rewriting git history does not remove a leaked credential from existing clones

A history rewrite creates new objects beside the old ones rather than editing them, and the old ones already sit on other people’s machines. The DESCRIPTION of git-filter-branch(1) says so in capitals: “WARNING! The rewritten history will have different object names for all the objects and will not converge with the original branch.”

Even the local copy survives. A few more steps are needed, says the same manual page’s checklist for shrinking a repository, “because Git tries hard not to lose your objects until you tell it to.” Run a routine gc without that checklist and git-gc(1) keeps the blob: --prune defaults to “2 weeks ago”, gc.reflogExpire to 90 days, gc.reflogExpireUnreachable to 30. The fortnight is deliberate, and it outlasts most incident bridges.

Hosted copies go the same way. Removing sensitive data from a repository lists where a commit stays reachable after a force push, “In any clones or forks of your repository / Directly via their SHA-1 hashes in cached views on GitHub / …”, then states the limit: “You cannot remove sensitive data from other users’ clones of your repository…” And its first instruction is what this article started from: “as a first step you need to revoke and/or rotate that secret.”

Workload identity federation swaps a 24-month credential for a 75-minute token

Workload identity federation lets the job present a token its own platform issued, and Microsoft Entra ID exchanges that for an access token with a 60-90 minute life. No long-lived string is pasted anywhere. The workload identity federation overview describes the trust it rests on: “Once that trust relationship is created, your external software workload exchanges trusted tokens from the external IdP for access tokens from Microsoft identity platform.”

Six moves, no stored password. The platform signs a token for the workload, which posts it with its client ID to the Entra token endpoint. Entra fetches the issuer’s OIDC metadata and signing keys, then matches three claims against the credential you registered: issuer, subject, audience. A token comes back. The job reads ADLS Gen2 until it expires.

Now derate the comparison as hard as the policy allows. Configurable token lifetimes bounds AccessTokenLifetime between “Minimum: 10 minutes (00:10:00)” and “Maximum: One day (23:59:59)”, one second short of 24 hours, so call the ceiling 1,440 minutes and let the rounding favour the attacker. Misconfigure the policy to it and the window goes from 75 to 1,440 minutes, a derating of 1,440 / 75 = 19.2 times, so the advantage falls from 576 to 576 / 19.2 = 30. It widens the other way: a secret at the 24-month maximum runs 2 × 365 = 730 days, or 731 if a leap day lands inside it, 730 × 24 = 17,520 hours, 17,520 × 60 = 1,051,200 minutes, 1,051,200 / 75 = 14,016 token windows. The decision flips nowhere in that range. Cryptography is never the hard part. The six-place inventory is, and it is where our data engineering work on a job like Kettlebridge’s spends its first week.

Count what workload identity federation costs before you adopt it

Twenty credentials per identity, no wildcards, every issuer-and-subject pair typed by hand. Workload identity federation for app considerations sets the bill: “A maximum of 20 federated identity credentials can be added to an application or user-assigned managed identity”, a 600-character limit on issuer and on subject, and a name of 3 to 120 characters that is immutable once created.

The cap derives a threshold rather than suggesting one. Issuer and subject together “must be unique on the app”, and the subject carries the namespace and service-account name, which differ per environment. One job needs one credential per environment, so J jobs across E environments need J × E registrations and the ceiling bites at J × E = 20. At two environments, J = 10. The eleventh job needs a second app registration, and the inventory stops being one list.

Then the failure modes you pay for in debugging time. A wrong subject is accepted in silence: the credential “is created successfully without error”, and “You won’t get an error, the exchange fails without error”. A correct one can still fail for its first few minutes, while the directory cache catches up. The propagation section says such a request “might fail with error message: AADSTS70021: No matching federated identity record found for presented assertion.” And creating several at once under one managed identity “triggers concurrency detection logic”, which Terraform’s azurerm provider did below version 3.40.0.

One thing I’d check in the tenant rather than the documentation. Microsoft publishes three figures for a long-lived continuous-access-evaluation token — “24-28 hours” on configurable token lifetimes, “20 to 28 hours” on access tokens, “up to 24 hours” on the workload-identity CAE page — and I can’t tell you which governs. It might never reach a Spark job: CAE there is “supported only on access requests sent to Microsoft Graph as a resource provider”, and “Managed identities aren’t supported.”

When a client secret is still the right answer

Inside Azure, with nothing running anywhere else, I wouldn’t register a single federated credential. A user-assigned managed identity removes the stored secret at zero registrations, zero issuer URLs and zero subjects to keep matched, and that work is the whole of federation’s setup cost. The crossover isn’t a count of jobs. It is whether one workload runs off Azure: the day a build agent or an on-premises Kubernetes cluster needs that storage account, the managed identity stops being reachable and the fleet moves across together.

What we insist on either way is small. Derive the rotation period from a stated tolerance rather than inheriting the 24-month ceiling, which gives 30 days by the argument above. Switch the history server’s cleaner on, because Spark’s monitoring documentation defaults spark.history.fs.cleaner.enabled to false, and spark.history.retainedApplications only drops applications “removed from the cache”, not from storage. And leave spark.redaction.regex where it is rather than growing it. A fix that exists in the documentation and a default nobody moved are two different things, which is the gap npm’s install-script change sits in too.