An AI assistant that can investigate a production incident across metrics, cluster state, and logs — and then propose a fix as a pull request. It holds no credentials to change anything itself. A person reviews the pull request, and the normal deployment pipeline applies it.
Platform team of six engineers and a manager, supporting four product teams and around 50 services. I owned this end to end — the capability split, the tool design, the credential scoping, and the safety model. In production since early 2026. The delivery pipeline it proposes into was already ours, shared across the team.
Triage on a production-like incident went from 11 manual steps to 4, and from roughly 22 minutes to 8.
one incident, timed both ways — the manual path across Grafana, kubectl, and logs, against the same investigation run through the tools. A single measured incident, not an average across many.
Remediation moved from "someone writes the manifest change by hand" to a reviewed pull request the agent drafts.
the mechanism is the claim here, not a hit rate. Every proposal is a pull request an engineer reads and decides on, so the number that matters is how many reach production without review — which is zero, by construction.
Two servers, split by what they are allowed to do. That split is the safety boundary — it isn't a deployment detail.
Ask the agent for something. Every request below is one the suite can receive; four of them it cannot carry out.
Pick a request to see which path it takes.
The decision that mattered was pull-request proposals over a direct apply tool. Direct apply is easier to build and much nicer to demo — you ask, the thing changes. But it needs live cluster write credentials, bypasses code review, and quietly creates a second deployment path alongside the reviewed one. That second path is the real risk: not that the agent is wrong, but that changes reach production by a route nobody watches.
The same reasoning shapes the tools themselves. Every parameter that policy should decide is decided in the server, not passed in by the caller — so the tool signature is the policy, and a compromised or misled client cannot widen it:
# The registry prefix and namespace are fixed here, not accepted as arguments.
# A caller chooses the image tag and nothing else.
NAMESPACE = os.environ.get("MCP_MUTATING_NS", "mcp-sandbox")
REGISTRY_PREFIX = os.environ["MCP_MUTATING_REGISTRY_PREFIX"]
# Rejects '/', ':' and '@' — so a tag can never rewrite the registry prefix
# into a different repository or pin a foreign digest.
TAG_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$")
@mcp.tool()
def create_test_pod(image_tag: str) -> str:
if not TAG_RE.match(image_tag):
return f"REFUSED: '{image_tag}' is not a valid image tag."
# serviceAccountName deliberately omitted, and the token automount
# disabled: the pod it creates holds no API credentials at all.
spec = V1PodSpec(automount_service_account_token=False, ...)Four read tools cover the investigation surface: a metrics query, workload status, cluster events, and pod logs bounded by namespace, a fifteen-minute window, and a line count. Logs are the widest data-exposure surface in the set, so they are the most tightly bounded.
The first version of the write tool accepted a repository path from the model.
It read naturally — the agent picks the file, the tool changes it — and would have worked for the cases I had in mind. It also meant anything able to influence the model's reasoning could point the tool at a file I'd never considered: a shared manifest, a CI workflow, the deployment config. Logs and metric labels are attacker-influenced data; I had built a path from that evidence to a file path with no trusted step in between.
I replaced it with a server-side map from a service name to the paths that belong to it. The model expresses intent — this service, this kind of change — and trusted code resolves what that means on disk. Anything outside the map is refused rather than negotiated.
That's the rule the rest of the design follows: the model expresses intent, trusted code resolves the target. It's why the registry prefix and namespace are pinned server-side rather than passed in, and why the delete tool refuses to touch anything it didn't create. A tool signature is a security boundary, and I'd written that one as a convenience.