Application credentials were maintained by hand as Kubernetes Secrets — base64 rather than encryption, and readable by anyone with namespace access unless the cluster is explicitly configured otherwise. I moved the source of truth into AWS Secrets Manager, where credentials can be audited and rotated centrally, and gave each workload a scoped identity that reads only its own. The Kubernetes Secret still exists — it is now generated at pod start instead of maintained by a person — which is exactly what let the applications keep using envFrom with no code change. All of it under production traffic.
Scope and ownership. Platform team of six engineers and a manager. I owned the consumer layer end to end: the inventory, the cutover sequencing, the per-app IAM scoping, the pattern itself, and the migration pull requests. I deliberately did not take ownership of the CSI driver install — it had been installed by hand from another workstream, and adopting it would have meant a second release owner on a live DaemonSet that another team's database already depended on. I documented who actually owned it instead.
The authoritative copy of every credential moved out of the cluster and into a store that can be audited and rotated, with a scoped IAM role per application rather than one shared credential.
Method: the security claim is structural rather than measured. Each app's role grants read on exactly one secret path, so the blast radius of a compromised workload is that app's own credentials — that's a property of the policy, checkable by reading it, not a number I collected.
Zero application code changes. Two production apps migrated on consecutive days with no rollback.
Method: verifiable in the migration pull requests — they touch manifests only, no application source. "Two apps on consecutive days" is a count of what shipped, not a benchmark. A third app was ruled out of scope before work started on it.
The CSI driver runs on every node and fetches secrets from AWS at pod start, authenticating through IRSA — so there is no long-lived credential anywhere in the path. Each application declares what it needs, and gets a service account annotated with a role scoped to just its own secret.
The detail that decided the whole design was how the applications actually consumed their configuration. Every one of them read config as environment variables through envFrom.secretRef; none mounted secrets as files. That matters more than it sounds, because the obvious way to use this driver is a volume mount — and a volume mount would have delivered files to applications that never read files. Nothing would have broken loudly. They'd simply have started with nothing.
So each declaration carries a secretObjects block that syncs the fetched values into a Kubernetes Secret using the key names the application already expected. That mapping is the entire reason no application code had to change:
# Redacted. The secret ARN and app name are placeholders.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
spec:
provider: aws
parameters:
objects: |
- objectName: "arn:aws:secretsmanager:<region>:<account>:secret:<app>-db"
objectType: "secretsmanager"
jmesPath: # pull individual keys out of the JSON
- path: '"username"'
objectAlias: "username"
- path: '"password"'
objectAlias: "password"
# The part that made this a zero-code-change migration: sync into a
# Kubernetes Secret under the SAME key names the app already read.
secretObjects:
- secretName: <app>-db-credentials
type: Opaque
data:
- key: username
objectName: username
- key: password
objectName: passwordThere's a trap in that arrangement worth naming, because it caught me. The CSI volume mount is required even when the application only uses envFrom and never reads the file — the mount isn't how the secret is delivered, it's what triggers the fetch at all. Leave it out and the driver never runs, the Kubernetes Secret is never created, and the application starts anyway reading empty environment variables. It's easy to reason your way out of including a mount the app demonstrably doesn't use.
I sequenced the cutover by ascending risk rather than by ticket order, starting with the application that already had its own service account and an existing secret — so the first migration tested the pattern with the fewest unknowns attached. Before any of it, a throwaway pod in its own namespace mounting a test secret proved the driver actually fetched. Doing three at once would have turned one lesson into three incidents.
Later, a credential rotation landed on one of these services — and this driver fetches a secret when a pod starts, not when the secret changes. A rotation only reaches a pod when that pod restarts.
A deploy went out shortly afterwards. The new pods couldn't come up: they failed their readiness probe, so the rolling update stopped partway through. That left the fleet split — some replicas still running on the old credential and serving traffic normally, others unable to start at all. Error rate climbed on the paths that had lost capacity, and the burn-rate alert fired before anyone reported it.
My first assumption was that the deploy was bad. That is what a dashboard suggests when error rate rises next to a deployment marker, and it was wrong — the deploy was a symptom. It just happened to be the thing that made pods restart and pick up the new value.
I stopped the rollout before I understood the cause. That kept the healthy replicas serving and stopped it getting worse, which bought the time to actually look.
The cause was the shape of the new value. Extraction is type-sensitive: a port stored as 3306 rather than "3306" fails the jmesPath lookup, so the Kubernetes Secret was never built, the environment variables were never populated, and readiness failed. Nothing in that path errors loudly — it quietly doesn't work. About twelve minutes of degraded capacity, roughly a quarter of that month's error budget.
Correcting the value took a minute. The change that mattered was validating a secret's shape before it can be rotated in, rather than discovering the problem whenever a pod next restarts — which could be days later, by which point nobody connects the two events.
I'd make the same design call again. The secretObjects mapping is what let production applications move without touching their code, and that was worth the sharp edges. What I'd change is how I verified: I watched pods come up healthy, and on this path healthy is exactly what the failure looks like. The check has to be that the credential arrived, not that the process started.