An AI assistant that can investigate a production incident across metrics, cluster state, and logs — and then propose a fix as a pull request. It holds no credentials to change anything itself. A person reviews the pull request, and the normal deployment pipeline applies it.
Scope and ownership. Platform team of six engineers and a manager, supporting four product teams and around 50 services. I owned this end to end — the capability split, the tool design, the credential scoping, and the safety model. In production since early 2026 — about six months. The delivery pipeline it proposes into was already ours, shared across the team.
Triage on a production-like incident went from 11 manual steps to 4, and from roughly 22 minutes to 8.
Method: one incident, timed both ways — the manual path across Grafana, kubectl, and logs, against the same investigation run through the tools. A single measured incident, not an average across many.
Remediation moved from "someone writes the manifest change by hand" to a reviewed pull request the agent drafts.
Method: the mechanism is the claim here, not a hit rate. Every proposal is a pull request an engineer reads and decides on, so the number that matters is how many reach production without review — which is zero, by construction.
Two servers, split by what they are allowed to do. That split is the safety boundary — it isn't a deployment detail.
The decision that mattered was choosing pull-request proposals over a direct apply tool. A direct-apply tool is easier to build and much nicer to demo — you ask, and the thing changes. But it needs live cluster write credentials, it bypasses code review, and it quietly creates a second deployment path alongside the reviewed one. That second path is the actual risk: not that the agent is wrong, but that changes start reaching production through a route nobody is watching. So the agent never got that power.
The same reasoning shapes the tools themselves. Every parameter that policy should decide is decided in the server, not passed in by the caller — so the tool signature is the policy, and a compromised or misled client cannot widen it:
# The registry prefix and namespace are fixed here, not accepted as arguments.
# A caller chooses the image tag and nothing else.
NAMESPACE = os.environ.get("MCP_MUTATING_NS", "mcp-sandbox")
REGISTRY_PREFIX = os.environ["MCP_MUTATING_REGISTRY_PREFIX"]
# Rejects '/', ':' and '@' — so a tag can never rewrite the registry prefix
# into a different repository or pin a foreign digest.
TAG_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$")
@mcp.tool()
def create_test_pod(image_tag: str) -> str:
if not TAG_RE.match(image_tag):
return f"REFUSED: '{image_tag}' is not a valid image tag."
# serviceAccountName deliberately omitted, and the token automount
# disabled: the pod it creates holds no API credentials at all.
spec = V1PodSpec(automount_service_account_token=False, ...)Four read tools cover the investigation surface: a metrics query, workload status, cluster events, and pod logs bounded by namespace, a fifteen-minute window, and a line count. Logs are the widest data-exposure surface in the set, so they are the most tightly bounded.
The first version of the write tool accepted a repository path from the model.
It read naturally — the agent decides which file to change, the tool changes it — and it would have worked fine for the cases I had in mind. It also meant that anything able to influence the model's reasoning could point the tool at a file I had never considered: a shared platform manifest, a CI workflow, the deployment configuration itself. Logs and metric labels are attacker-influenced data; treating them as evidence is fine, but I had built a path from that evidence to a file path with no trusted step in between.
I replaced it with a server-side map from a service name to the paths that belong to it. The model expresses intent — this service, this kind of change — and trusted code resolves what that means on disk. Anything outside the map is refused rather than negotiated.
Structurally, that is the rule the rest of the design follows: the model expresses intent; trusted code resolves the target. It's why the registry prefix and namespace are pinned server-side rather than passed in, and why the delete tool refuses to touch anything it did not create. I would not make the original call again — a tool signature is a security boundary, and I had written that one as a convenience.