Dev.to · 3 min read

I recorded my Kubernetes AI agent failing, on purpose

I recorded my Kubernetes AI agent failing, on purpose

Most demos of AI-for-infrastructure tools show a clean run. I think that is exactly backwards, and I want to explain why by walking through the thirty seconds of my own demo that a normal product video would have cut. The scene A deployment is crash-looping. The agent has already diagnosed it. I ask it to restart the deployment. It stops and asks for approval. I approve. The restart runs, and the command succeeds. Then I ask the follow-up question: did the restart change anything? No. The pods fail with the same error. A restart was never going to supply a missing environment variable. The agent re-reads the pods, re-reads the events, and names the same root cause it found at the very beginning: DATABASE_URL is not set, so the container exits 1. The restart was a reasonable thing to try and the wrong thing to fix it, and the system is the one saying so. Why that scene is the product The worst failure for an incident-response tool is not being unhelpful. It is emitting something that looks like a real diagnosis and is not. At 3am a confident wrong answer costs more than no answer, because it sends a tired human down a wrong path with false authority. So the design follows from that. The approval gate is at the tool boundary, not in the prompt. A guardrail written into a system prompt is a suggestion, and models can be argued out of suggestions. Instead every mutating operation passes through one chokepoint that returns exactly three answers: do it, ask a human, or refuse. The model does not get a vote, so there is nothing to talk around. Detectors are compiled predicates, not model calls. They are always on, they run on every observation, and they cost zero tokens. The LLM is only invoked once a detector actually fires. Watching your cluster is free; thinking about it is the expensive part, so it happens last. Every decision is appended to a hash-chained log. A run can be replayed afterwards instead of remembered. If you are going to let software touch production, "what exactly did it do, and why" has to be answerable after the fact. Roles are real. readonly, operator, admin and superadmin, enforced per key. What it reads kubectl for cluster state, Prometheus for metrics via PromQL, and Loki for logs via LogQL. It answers in plain English and quotes the evidence it actually read, so you can check it rather than trust it. The video Eight minutes, every terminal scene a verbatim recording against a live cluster. Nothing typed by hand, nothing reconstructed. The approval-gate segment starts at 2:27. Try it In your browser, nothing to install: https://kubeintellect.com/demo pip install kubeintellect - AGPL-3.0, self-hosted, bring your own LLM provider Code: https://github.com/MSKazemi/kubeintellect The v1 architecture is written up in the Journal of Grid Computing (https://doi.org/10.1007/s10723-026-09837-6), with a preprint at https://arxiv.org/abs/2509.02449. The repo is several generations past that now. If you run Kubernetes in production, the thing I most want to hear is which failure you would throw at it that I have not handled.

This is a summary aggregated from Dev.to. Read the complete article on the original site:

Read full article at Dev.to

More AI & Machine Learning News