Operate
Reliability is the reason operators will or will not let an agent near real work.
1 in 3
structured benchmark attempts still fail, even after OSWorld jumped from 12% to 66% task success.
Stanford HAI, AI Index 20261A system that is impressive on Tuesday and quietly wrong on Thursday will be shut off. COOs do not need brilliance. They need dependable completion.
How I see it
AI agent reliability
Agent reliability is not model IQ. It is whether the system completes the same class of case the same way, fails in visible ways, and does not invent a finished state.
The common failure modes are specific: missing context, stale records, tool errors, over-confident write-back, retry storms, and policy drift after a prompt change. Each one needs a designed response.
Retries are not a reliability strategy by themselves. They can multiply cost and duplicate actions. Fallbacks should move the case to a person or a safe hold, not to a more creative guess.
If you cannot describe how the agent fails, you are not ready to put it on a customer, finance, or operations path.
Common mistakes
What teams usually get wrong.
Retry until it works
Unbounded retries create duplicate updates and surprise bills.
Hiding uncertainty
An agent that always sounds sure trains people not to look.
Changing prompts without a regression set
A small wording change can move a reliable agent into a new failure mode.
A useful diagnostic
Five questions before you fund the work.
What are the top three ways this agent can be wrong?
If the team cannot name them, they have not operated it yet.Is there a maximum retry and a maximum spend per case?
No ceiling means reliability work will show up on the invoice.Does a tool failure become an exception or a guess?
Guessing is how silent failures start.Can you roll back or flag a bad action?
Reliability includes recovery, not only prevention.Do prompt and model changes require a re-score?
If not, reliability is accidental.
Economic model
Reliability gatecritical-miss rate near zero + known failure modes + bounded retries + fallback path
Average quality can look fine while a small class of cases creates all the damage.
Three credible paths
How far should you go?
Do not force one solution. Choose the path the economics, the risk, and the organization can support.
Hold and escalate
On uncertainty or tool failure, stop and give a person a complete packet.
The agent is new or the action is hard to reverse.
Throughput depends on the exception queue.
Bounded retry then hold
Retry only idempotent reads or safe checks, then escalate.
Transient system issues are common and writes are protected.
Someone must define which actions are safe to retry.
Self-check then act
Add verification steps before write-back on the common path.
You can check the result against a system of record or a rule.
Checks add cost. They have to pay for themselves in avoided misses.
When this is the wrong next step
Do not fund an agent here.
- The company wants the agent live before anyone will name the failure modes.
- There is no fallback process, so a model outage stops the work.
- The workflow cannot tolerate even a small class of silent errors.
A useful next step
Bring one workflow. Get guided into production.
We guide the implementation, go deep on the technical path, and stay hands-on through operations — or tell you when a simpler answer is better.
Discuss an AI opportunity