Kagent on Kubernetes: What Does it Give Your AI Platform?

Table of Contents Introduction An Agent CR becomes a running agent What actually installs What kagent owns above the pod Kagent declarative vs BYO agent — 2 ways to own the loop Agent call request flow The frontier: isolation 1. Introduction K agent is an open-source Kubernetes operator for AI agents. It allows you to write a custom resource describing what the agent should be, apply it, and a controller turns it into a running workload. Thirty lines of YAML, and you have an agent in a pod. I ran a small agent fleet on EKS with kagent to experience for myself its capabilities, learn more AI platform engineering and to properly understand the boundaries of what it provides. The next question is then, what was just put in your cluster, and what else is there to do? An agent is a long-lived workload that talks to a model, calls tools that touch real systems, holds state, and possibly+probably talks to other agents. Some of these surfaces the operator now owns for you while it deliberately doesn’t own others. Here’s a map for illustration: In short: kagent owns the workload, the reasoning loop, tool registration, session state. You still own namespaces, network policy, secrets, identity, etc. The same platform work as everything else you run. The operator does not own a clean bottom half of the stack. Each layer is divided, and the right-hand side is ordinary platform work — quota, egress, secrets, RBAC, backup, identity. 2. An Agent CR becomes a running agent The core of kagent is the ability to describe an agent as a Kubernetes object. A Go controller reconciles it into the actual workload. Its manifest would look something like this: apiVersion: kagent.dev/v1alpha2 kind: Agent metadata: name: cluster-diagnostics namespace: kagent spec: type: Declarative declarative: modelConfig: default-model-config systemMessage: | You are a read-only Kubernetes troubleshooting specialist… tools: - type: McpServer mcpServer: kind: RemoteMCPServer name: kagent-tool-server toolNames: [k8s_get_resources, k8s_get_events, k8s_get_pod_logs] Apply that and the controller produces a Deployment, a Service, a ServiceAccount with the RBAC that workload needs, and a Secret holding the rendered agent config. With that, Agents have become declarative Kubernetes objects that can be versioned and rolled out with GitOps. 3. What actually installs Here’s what’s in my kagent namespace. Here a single namespace is used for ease of exploration and illustration. *This isn’t a stock install, took the command output after switching off several built-in agents and adding a few custom agents and MCP servers as i was exploring.* As can be seen, there’s an operator, a database, a tool server, a web UI, and a set of prebuilt agents that work the moment the kagent chart lands — `k8s-agent`, `helm-agent`, `istio-agent`, `promql-agent` and others. $ kubectl -n kagent get pods kagent-controller-cf74f9f96-kwblb 1/1 Running # the operator kagent-postgresql-6c47c5bc6f-hg6cz 1/1 Running # state store kagent-tools-56494b5564-7zsdn 1/1 Running # built-in MCP tool server kagent-kmcp-controller-manager-76bb479b6-4s8z4 1/1 Running # kmcp — build/deploy your own MCP servers kagent-ui-69cf9cd7cf-rpbmr 1/1 Running # web UI kagent-default-67c785f9db-wnzc2 1/1 Running # part of the kagent install helm-agent-5fd78944d8-wzdkx 1/1 Running # built-in agent istio-agent-6dfb7b5f-xgm6v 1/1 Running # built-in agent k8s-agent-9f9548bdc-mnrdb 1/1 Running # built-in agent promql-agent-f7cb48786-tf5jb 1/1 Running # built-in agent cloud-diagnostics-5cf9fc9684-ppg6v 1/1 Running # mine cluster-diagnostics-756cf6455b-vdpgz 1/1 Running # mine cluster-remediation-76ccf4f698-xf99n 1/1 Running # mine incident-commander-5c89f99d6d-rrpdp 1/1 Running # mine (orchestrator) investigation-loop-797bbc9f7c-kp87w 1/1 Running # mine (BYO) cost-sentinel-644f7c6c44-kwxss 1/1 Running # mine aws-documentation-76b96d6c8f-tgjzj 1/1 Running # mine (MCP server) aws-eks-57977cb77d-mhgbq 1/1 Running # mine (MCP server) aws-pricing-9576cf679-m9722 1/1 Running # mine (MCP server) agent-sandbox-probe 1/1 Running # mine (sandbox experiment) 4. What kagent owns above the pod Looking at the API surface itself (by k8s API group), kagent comes with 9 custom resources under 1 API group. Each is a thing that used to live inside an agent application when needed; and now they live in the cluster API. $ kubectl get crd -o custom-columns=NAME:.metadata.name,GROUP:.spec.group --no-headers \ | grep 'kagent.dev$' | sort agentharnesses.kagent.dev kagent.dev agents.kagent.dev kagent.dev mcpservers.kagent.dev kagent.dev memories.kagent.dev kagent.dev modelconfigs.kagent.dev kagent.dev modelproviderconfigs.kagent.dev kagent.dev remotemcpservers.kagent.dev kagent.dev sandboxagents.kagent.dev kagent.dev toolservers.kagent.dev kagent.dev ``` `ModelConfig` and `ModelProviderConfig` hold the provider, the model, and the credential reference. Agents can then point to them by name (refer to yaml above). Change a model, or move a model provider’s key, and you’re editing a namespaced object without needing to redeploy agents. (But of course, the fact that prompts and tool-calling behaviour are model-specific needs to be accounted for when changing this shared object.) While some of what kagent owns is a CRD you create, some are just a field on the Agent you already have. For examplerequireApproval allows for human-in-the-loop and is nested inside the tool reference: $ kubectl explain agent.spec.declarative.tools.mcpServer FIELDS: allowedHeaders <[]string> apiGroup kind name -required- namespace requireApproval <[]string> 5. Kagent declarative vs BYO agent — 2 ways to own the loop kagent runs two kinds of agent. The CR spec.type is an enum with exactly two values, Declarative and BYO. Declarative agents use kagent’s engine, you supply a system prompt, a tool list, a model config and the ADK (Google’s Agent Development Kit framework) owns the model calls, tool dispatch, retries and context handling. A capable multi-tool agent can be just ~30 lines of YAML or less. Kagent ships two runtime implementations of pythonand go, selectable per agent. Both run the agent as an HTTP service and speak the same protocols. BYO (bring-your-own) agents replace the engine. You ship a container that implements the loop yourself; kagent deploys it and routes messages to it. One of mine, investigation-loop, is a BYO LangGraph StateGraph. Just to illustrate a comparison point on looping, here’s pseudo code: # BYO - you own the loop. graph.add_node("gather_evidence", gather_evidence) graph.add_node("hypothesize", hypothesize) graph.add_node("verify_hypothesis", verify_hypothesis) graph.add_node("conclude", conclude) graph.set_entry_point("gather_evidence") graph.add_edge("gather_evidence", "hypothesize") graph.add_edge("hypothesize", "verify_hypothesis") graph.add_conditional_edges( #decided by code, and not by the model "verify_hypothesis", should_continue, {"conclude": "conclude", "gather_evidence": "gather_evidence"}) graph.add_edge("conclude", END) A Declarative agent loops too without a custom build; in fact every tool-calling agent does. What the BYO agent changes is who decides the branch. In a Declarative agent, “should I investigate further or answer now?” is a judgement the model makes inside its loop, shaped by the prompt. In the code above, it’s should_continue, a Python function that can be unit-tested, with a termination condition that can be asserted on and an iteration count that can be bound in code. Something still has to act on the decision each time regardless whether its dispatching the tool, feeding the result back or stopping a loop that won’t converge. That’s the ADK, and we can see the difference between the declarative agent (manifest shown in section 2) and BYO agent by checking their image: #Declarative - don't have to supply container image $ kubectl -n kagent get deploy cluster-diagnostics -o jsonpath='{..image}' cr.kagent.dev/kagent-dev/kagent/app@sha256:d4be3183... #BYO - custom image $ kubectl -n kagent get deploy investigation-loop -o jsonpath='{..image}' .dkr.ecr.ap-southeast-1.amazonaws.com/aria/investigation-loop:latest A Declarative agent’s reasoning runs inside an image kagent publishes and reinherits on every upgrade. A BYO agent’s runs inside yours. What other differences are there? $ kubectl explain agent.spec.byo FIELD: byo DESCRIPTION: BYO configures a "bring your own" agent backed by a user-provided container image. Kagent deploys the image and expects it to serve the agent over the A2A protocol on port 8080. Required if type is BYO. FIELDS: deployment The BYO CR only offers a single field = deployment. Meanwhile modelConfig, tools, memory, context, and the approval gate all live under spec.declarativeand aren’t available for BYO. So, much of the abstraction benefits mentioned earlier only apply to declarative agents. I would say that a good decision guideline would be to go declarative unless you can name what’s unusual about your loop. Additionally, I would say, (as someone exploring platform work and not a dedicated AI app/agent developer for now), declarative agents can be really useful for iterating and building infra ops related agents. Here’s the current inventory of my platform in progress: $ kubectl -n kagent get agents -o custom-columns=NAME:.metadata.name,TYPE:.spec.type NAME TYPE cloud-diagnostics Declarative cluster-diagnostics Declarative cluster-remediation Declarative cost-sentinel Declarative deploy-diagnostics Declarative helm-agent Declarative incident-commander Declarative investigation-loop BYO istio-agent Declarative k8s-agent Declarative promql-agent Declarative 6. Agent call request flow The model<->agent cycle is not kagent-specific since every agent framework has it. The tool call is a network hop to a separate workload (an MCP server, another agent, etc.) with its own credential, rather than an in-process function; and the controller is an optional front door, not a hop. Every agent has its own Service, agent-to-agent calls go pod to pod, and keeping request traffic off the controller keeps the dataplane decoupled from reconciliation. The number of round trips is decided by the model at request time, which is where cost and latency actually come from. This diagram was easily mapped since calling my incident commander agent returned a task object whose history[] records every step it took. What happened was the t he model asked for a tool instead of answering. It emitted a function_call — name: kagent__NS__cluster_diagnostics, with an args.request spelling out what it wanted to know. The call travelled as A2A, pod to pod, and landed on the cluster diagnostics agent. Sub-agents are declared in the very same tools array as MCP servers, as peers (refer to the declarative agent manifest from section 2 for how the configuration looks like). Let’s double check: $ kubectl -n kagent get agent incident-commander -o jsonpath='{.spec.declarative.tools}' [{"type":"Agent","agent":{"kind":"Agent","name":"cluster-diagnostics","namespace":"kagent"}}, {"type":"Agent","agent":{"kind":"Agent","name":"cloud-diagnostics", "namespace":"kagent"}}, {"type":"Agent","agent":{"kind":"Agent","name":"cost-sentinel", "namespace":"kagent"}}, {"type":"Agent","agent":{"kind":"Agent","name":"investigation-loop", "namespace":"kagent"}}] You can see from above that sub-agents are a tool type at kubernetes CRD-level. The `subagent’s` answer came back as a function_response — its own token usage attached: 3,588 prompt tokens, just for that one sub-question. With that result now in context, the orchestrator's model was called once more, and this time it wrote text instead of asking for another tool. That's what ends the loop — not a signal, just the absence of another function_call. Task state: completed. How agents reach tools, and each other Two protocols carry everything else an agent does outside its own process. MCP (Model Context Protocol) reaches tools. kagent ships MCP servers for Kubernetes, Helm, Istio, Argo, Prometheus and more, plus kmcp can be used for building your own. Tool access itself is a manifest /CR of MCPServer for those you run, andRemoteMCPServer is for registering an endpoint not run by you. A2A (Agent-to-Agent) reaches other agents (i.e. the delegation to cluster-diagnostics above). It’s JSON-RPC over HTTP, and the protocol supports streaming via SSE (though this call didn’t use it). It went direct pod-to-pod over Service DNS, not relayed through the controller. Agent Harness is a third newer surface that kagent carries. It is a long-running remote environment for sitting down and working with an agent (kagent's docs frame it around coding agents such as OpenClaw or Hermes) rather than calling it over an API. It always runs on Agent Substrate (a sister project on agent and execution isolation). 7. The frontier: isolation kagent reaches toward one more layer the original operator explicitly left alone, which is where the agent actually runs at the kernel level. There’s a resource for it calledSandboxAgent and it's a full peer of the regular Agent CR. The sandboxAgent carries its own type, declarative, byo. Which isolation backend runs it is decided by which config block you fill in, not an enum: $ kubectl get crd sandboxagents.kagent.dev \ -o jsonpath='{...spec.properties.sandbox.description}' Sandbox configures sandboxed execution behavior shared across runtimes. This is intended for sandboxed declarative execution today, and can also be consumed by BYO agents. $ kubectl get crd sandboxagents.kagent.dev \ -o jsonpath='{...spec.properties.substrate.description}' Substrate is optional Agent Substrate-specific settings. Looking at two separate projects that SandboxAgent can point at, Agent Substrate is kagent’s own family, same Linux Foundation project umbrella, a first-class page in kagent’s core-concepts docs, and the exclusive runtime for AgentHarness. Agent Sandbox is a Kubernetes SIG Apps subproject , its own controller, its own CRDs — installed and operated entirely separately, and absent from kagent's own documentation despite being a real, working option. Whichever you pick, the cost is the same shape: a second control plane to run and upgrade on top of the one you already have. Kagent on Kubernetes: What Does it Give Your AI Platform? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.