Operate
If you cannot see a case move through the agent, you cannot operate it.
Logs that only a researcher understands are not observability. Operators need to know what the agent did, what it cost, and which cases are stuck.
89%
of teams can observe their agents. Only 52% evaluate whether the work is right.
LangChain, State of Agent Engineering 20261
How I see it
AI agent observability
Agent observability has to answer production questions. Which cases completed? Which escalated? Where did a tool fail? What did the run cost? Did quality move after yesterday’s change?
Traces should follow the case, not only the model call. A COO does not need token charts. They need volume, cycle time, exception rate, cost per completion, and a way to inspect a bad case without a war room.
Observability is also how you keep the system honest. If write-back happened, it should be visible in the same tools the team already uses, with a link back to the agent’s evidence.
Without this, every incident becomes a debate about whether the model is “getting worse.”
Common mistakes
What teams usually get wrong.
Model metrics without case metrics
Token use can look healthy while customers wait and records are wrong.
Unreadable traces
If only the original builder can debug a case, the agent is not operable.
No cost signal
A quiet increase in retries can erase the business case.
A useful diagnostic
Five questions before you fund the work.
Can an operator find a single case and see every action?
If not, you have logs, not observability.Are completion, escalation, and failure counted separately?
Mixing them hides the only numbers that matter.Is cost per case visible daily?
Weekly invoice surprises are too late.Do you get an alert on critical-miss patterns?
Reading dashboards after the fact is not enough.Can you compare this week to last week after a change?
If not, you cannot tell improvement from drift.
Economic model
Operating picturevolume + completion rate + exception rate + cost per case + cycle time
These five numbers are enough for an executive review. Everything else supports them.
Three credible paths
How far should you go?
Do not force one solution. Choose the path the economics, the risk, and the organization can support.
Case log and weekly review
Record each run and review exceptions once a week.
Volume is still low and one owner can read every miss.
Will not catch a fast-moving failure.
Live case traces and cost
Trace every case and watch cost and exceptions daily.
The agent is on a real operating path.
Someone has to look at the signals.
Alerts and regression views
Alert on critical patterns and compare quality after every change.
Multiple agents or higher volume make manual review too slow.
Alert noise will train people to ignore the system if thresholds are sloppy.
When this is the wrong next step
Do not fund an agent here.
- Nobody will own the dashboard after the project team leaves.
- The company refuses to store traces because it has not decided a retention policy, and still wants production.
- The only available view is a vendor token graph.
A useful next step
Bring one workflow. Get guided into production.
We guide the implementation, go deep on the technical path, and stay hands-on through operations — or tell you when a simpler answer is better.
Discuss an AI opportunity