← Projects

Taking the long-lived AWS keys away

What it is

Build pipelines and applications used to authenticate to AWS with access keys — long strings that work forever, sit in a settings page or a Kubernetes Secret, and are just as useful to anyone who copies them. I replaced them with identity: the pipeline and the workload each prove who they are at the moment they run, and get a credential that expires in minutes. Over 120 standing keys stopped existing.

Scope and ownership. Earlier role, late 2021 into 2022 — this work began as GitHub Actions introduced OIDC federation to AWS, which is what made it possible. Two of us on the DevOps side, supporting four application teams. I owned the identity design and the migration: the trust policies, the per-workload roles, the rollout sequence, and the inventory work that told us what was actually using each key. Every team that shipped through those pipelines had to change how their workflow authenticated, so the delivery was as much coordination as configuration.

What changed

120+ static access keys eliminated, across the account estate.

Method: a count of active access keys in the AWS IAM credential report before the rollout, summed across accounts. The credential report is the artifact — it's the natural thing to have pulled, and it's the same report anyone auditing this would ask for.

35 services moved onto per-workload identity via IRSA.

Method: a count of workloads making AWS API calls, taken from the service inventory. Not every service needed a role — plenty never called AWS at all. That's what makes 35 a real number rather than a suspiciously round one.

The middle row is the one that matters. Everything else follows from a credential that stops working on its own.

The security change underneath the counts is that a credential stopped being a thing that could be copied. A leaked access key works until somebody notices and revokes it, which historically means it works for a long time. A federated credential is minted for one run or one pod and expires on its own, so the window where a copy is useful is measured in minutes.

How it works

Two paths, the same underlying idea: the thing that wants access presents a signed token proving what it is, and AWS decides whether that identity may assume a role. Nothing is stored.

Two federated paths replacing stored keys A GitHub Actions workflow requests an OIDC token from GitHub and presents it to AWS STS, which validates it and returns temporary credentials. Separately, a Kubernetes pod presents its service account token to STS, which validates it against the cluster OIDC provider and returns temporary credentials scoped to that workload's role. In both paths nothing long-lived is stored anywhere. CI workflow no stored secret pod service account GitHub OIDC token scoped to repo + branch projected SA token cluster OIDC provider AWS STS checks aud + sub role minutes
Both paths end at the same question: does this identity, proven right now, get to assume this role?

For pipelines, the workflow asks GitHub for a token and hands it to AWS. The only thing the repository holds is the name of a role — not a credential:

permissions:
  id-token: write        # lets the job request an OIDC token
  contents: read

# ...

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: ${{ vars.IAM_ROLE }}   # a role ARN, not a key
    role-session-name: GitHubActionsRoleSession
    aws-region: ${{ env.AWS_REGION }}

For workloads, the pod presents its service account token and the trust policy decides. This is where the care goes, because a trust policy is the actual security boundary — the permissions attached to a role only matter once you've decided who may assume it. Two conditions have to be pinned: the audience, and the subject naming the exact service account.

# Audience: the token must have been minted for STS.
condition {
  test     = "StringEquals"
  variable = "${local.oidc_issuer}:aud"
  values   = ["sts.amazonaws.com"]
}

# Subject: this ONE ServiceAccount, in this ONE namespace.
# Anything looser — a wildcard, or only the namespace — lets every
# other pod in that namespace assume the role too.
condition {
  test     = "StringEquals"
  variable = "${local.oidc_issuer}:sub"
  values   = ["system:serviceaccount:<namespace>:<serviceaccount>"]
}

The subject condition is the line worth reading twice. Written with a wildcard it still works — tests pass, the pipeline goes green — while quietly granting the role to every pod that shares the namespace. That asymmetry is why this work rewards being slow: a leaked key eventually announces itself, but a trust policy that is broader than intended looks exactly like one that isn't.