Monitoring for a Kubernetes platform running about fifty services. Before it existed, nobody could answer "is the platform healthy right now?" without opening several tools and forming an opinion. Afterwards there was a single number, agreed in advance, that anyone could look at — and alerts that fired when that number was genuinely at risk rather than whenever a graph spiked.
Scope and ownership. Platform team of six engineers and a manager. Three environments, 40–50 services across 10–15 namespaces, 20–30 nodes, 15+ production deploys a day. I owned this one end to end — the Helm chart, the Prometheus configuration, the SLO definition, the dashboards, the alerting rules, and the runbooks. The 99.9% target has held since early 2024 — two and a half years. The IAM and networking it sits on were shared team ground.
Median time to resolve a sev-2 went from about 45 minutes to about 15.
Method: the median — not the mean — of sev-2 resolution time, comparing the quarter before burn-rate alerting to the quarter after, measured from impact rather than detection. A mean would let one long incident tell the whole story. It's one quarter against another rather than a controlled experiment, and there aren't many sev-2s in a quarter, so it's directional rather than precise.
Alert volume fell by roughly half, without removing coverage.
Method: alerts actually delivered by Alertmanager per week, counted before and after. The reduction came from changing what the rules ask, not from deleting rules — the burn-rate conditions still catch the incidents the old thresholds caught. That last part was my judgement call at the time, not something I proved with data.
The platform got its first error budget: about 43 minutes of unavailability per 30 days, visible to whoever was shipping.
Method: arithmetic, not measurement. 0.1% of a rolling 30-day window is 43.2 minutes.
The honest shape of the MTTR number is that almost all of the improvement was in noticing and in knowing where to look. Rollback had always been quick. What changed is that an alert started meaning something specific, and it went to the team that owned the service instead of a channel everyone had learned to ignore.
The clearest evidence that it worked isn't on this page. A credential rotation later took out part of a service's capacity, and the burn-rate alert fired before a single person reported it — which is the entire argument for this kind of alerting, made by something other than me. That incident is written up here.
Prometheus scrapes what the cluster already emits — ingress traffic, container metrics, node and control-plane state, and application metrics — on a 30-second interval with 30 days of retention. Grafana reads Prometheus and CloudWatch side by side, authenticating to AWS through IRSA so there are no long-lived credentials in the path. Alertmanager routes by severity and owning team.
Running Prometheus ourselves rather than buying a vendor was the decision worth defending. The signals already existed in Prometheus format — the ingress controller, kube-state-metrics and cAdvisor all expose it natively — so self-managing meant scraping what was already there instead of paying per host to re-ingest it. It also kept retention and cardinality in our hands, which on high-cardinality Kubernetes metrics is exactly where the cost surprises live. The trade is real, and we paid it: we carried the operational burden of the stack ourselves.
Availability is measured over a rolling thirty-day window, counting 5xx as failure and excluding 4xx as client error:
# 30-day availability. 5xx counts as failure; 4xx is treated as client error.
# Namespace redacted.
sum(increase(nginx_ingress_controller_requests{
namespace="<app>", status!~"5.."
}[30d]))
/
sum(increase(nginx_ingress_controller_requests{
namespace="<app>"
}[30d]))
* 100Running the stack ourselves has an operational cost, and it arrived during a staging rollout when Alertmanager went into a crash loop — the monitoring failing while it was meant to be watching everything else. The base chart enabled high-availability gossip unconditionally and staging ran a single replica, so it started up hunting for peers that did not exist. IPv6 dual-stack made the failed peer resolution surface immediately, which is why it looked like a networking problem for the first while; it wasn't, it just made the real fault visible faster. The chart fix was small — inject peer arguments only when replicas exceed one — and the change that mattered was a config-validity check in CI, so a rendered config that cannot start now fails before merge instead of at rollout.
The SLI counts some of our own failures as successes, and I shipped it that way on purpose.
Look at the artifact above: it excludes everything that isn't a 5xx. Treating 4xx as client error is the standard simplification and it's right for most of the class — a 404 for a URL someone mistyped is not a reliability failure, and counting it as one would make the number meaningless in the other direction.
But a 429 is a 4xx, and a 429 is us. When the platform is under enough load to start rejecting requests, those rejections are our failure and the SLI scores every one of them as a success. Which means the measurement is most optimistic exactly when the platform is least healthy — the error is not random, it's biased in the worst available direction. An SLO that looks best under stress is the specific way a reliability target can lie to you.
I'd ship the same v1 again, and that isn't a dodge. The platform had no reliability target at all, and a defensible simplification that exists beats a perfect definition still being argued about — you cannot tune alerting against a number nobody has agreed on. What I would change is treating the exclusion as finished rather than provisional. The correction is specific and I can describe it: re-include platform-generated 4xx — our own 429s and any 4xx produced by the ingress rather than the application — while keeping genuine 400s, 401s and 404s out. I knew the bias, and I should have written it into the SLO document next to the target instead of carrying it in my head.