Improve your agents

Launch is the midpoint, not the finish. The deployments that succeed are the ones somebody watches and improves. Kata CI gives you three loops, from cheapest to deepest.

Loop 1: feedback

Users rate responses in Chat and on Case tasks. This is your earliest and cheapest signal. The platform can go further than reading it: feedback is analyzed for patterns, patterns become proposed behavior rules, a human reviews and approves them, and approved rules apply to future runs.

Corrections become improvements without a single prompt edit, and nothing changes behavior without review. Watch for clusters: one thumbs-down is an anecdote, five on the same kind of request is a work item.

Loop 2: observability

When you need to know what actually happened, open Console → Observability. The Traces view streams every run in the workspace, chat and Case fills alike, with a stat strip on top: run counts, success rate, what's running now, and median latency.

Open any run and you get its full execution: model calls, knowledge retrievals, tool calls with inputs and outputs, timing, and cost.

The Console home rolls the same signals up to workspace level: agent and Case run volume, cost over time, top agents by usage, model spend breakdown, and knowledge health, filterable by time range.

Use traces to answer questions feedback can't:

• "Why did it answer that?" Check what retrieval returned; the wrong passage explains the wrong answer.
• "Why was it slow?" The timeline shows which step ate the time.
• "What is this costing?" Usage and cost roll up per agent and per workspace.

Loop 3: evaluation

Feedback tells you something is off; evaluation tells you whether the agent is good. Console → Eval gives you:

• Review queues where human reviewers score sampled real conversations against your quality criteria.
• Failure categories so recurring problems get named, counted, and tracked over time instead of rediscovered.
• Automated judges, calibrated against human reviews, that score at scale between human passes.
• The publish gate, which runs evaluation on every publish so regressions surface before users find them.

Where fixes go

The diagnosis decides the fix, and picking the right layer keeps agents maintainable: