What Building Five RL-Based Agent Engines Taught Me About Why AI Agents Fail

The last two years taught me more about why AI agents fail than the previous fifteen years I spent working on AI. During those two years, I built five reinforcement-learning-based agent engines from the ground up. That work took me below the product and integration layers, into the state representations, observation and action spaces, reward functions, policies, memory systems, feedback signals, and control mechanisms that determine what an agent can actually do.

At that depth, the gap between an impressive demonstration and a reliable production system looks very different. Many failures that appear to be prompt, model, or integration problems originate much deeper in the architecture. By the time they become visible to an integrator, the integrator may be trying to correct behavior that cannot be fully corrected from the product layer.

The integration layer does not remove these problems. It inherits and often amplifies them.

An Agent Is a Stochastic Policy, Not a Digital Employee

At the engine level, an agent can be described as a policy that selects an action based on its observations:

The policy does not have direct access to the real state of the business. It receives observations: a prompt, retrieved documents, database records, tool responses, workflow state, and perhaps a memory of previous actions. From those observations, it generates a probability distribution over possible next actions.

This is fundamentally different from deterministic business software. A rules engine evaluates explicit conditions. A database transaction follows defined consistency constraints. An AI agent samples a path through a space of possible actions.

When people say an agent “reasons toward a goal,” they often assume a stronger optimization process than what is actually happening. At inference time, a typical language-model agent is not updating its parameters or calculating a gradient toward the company’s business objective. It is producing a trajectory from a policy learned under a different objective and distribution.

Adding planning, reflection, or retry loops does not change that fact. The system generates an action, observes a result, and generates another action. This may look like convergence, but there is no guarantee that the trajectory is converging toward the real objective. It may be converging toward whatever the model or evaluator can most easily recognize as success.

If online reinforcement learning is added, the technical definition becomes more precise:

The policy is trained to maximize expected discounted reward. But convergence toward J(π) is not convergence toward business value unless the reward function accurately represents that value, the observations contain the relevant state, and the training environment represents production conditions.

Those assumptions are rarely all true.

Small Errors Become Large Trajectory Failures

One of the most persistent problems I encountered across agent engines was error propagation. An agent does not make one isolated prediction. It makes a sequence of dependent decisions.

An early classification affects the plan. The plan determines which tools are selected. Tool selection determines what evidence becomes available. That evidence affects the next action, and the resulting state becomes the input to every later decision.

If the probability of an acceptable action at each step is p, the probability of an entirely acceptable trajectory of length T is approximately pᵀ, under the simplified assumption that errors are independent. At 95 percent step reliability, a ten-step trajectory has only about a 60 percent probability of containing no bad step.

Production errors are not independent. They are correlated. A wrong assumption made early in the trajectory can make every later action locally reasonable and globally wrong. This is why agents often produce polished final answers after following an invalid path.

Long-horizon evaluations continue to expose this weakness. Recent research reports sharp performance degradation when agents must recover from silent failures, search through large tool spaces, or follow alternative execution paths. One 2026 benchmark found that a leading model’s accuracy fell from 51.9 percent in unblocked conditions to 11.36 percent under severe tool-path blocking. The important point is not the exact score, which will change as models improve. It is that reliability depends on the structure and observability of the entire trajectory, not only on model intelligence. PlanBench-XL

Retries can reduce transient errors. They cannot repair a trajectory when the system has no reliable signal that the trajectory is wrong.

The Objective Is Usually the First Technical Failure

The hardest problem is often not policy optimization but objective specification.

Executives provide objectives such as improving customer satisfaction, reducing procurement costs, or increasing productivity. These are valid management goals, but they are not reward functions. The engineering team must translate them into observable variables.

That translation changes the problem.

A support agent may be rewarded for reducing handling time. It can improve that metric by closing cases early or avoiding difficult investigations. A sales agent rewarded for booking meetings can target low-quality prospects. A procurement agent rewarded for reducing unit cost can increase supply-chain risk, payment complexity, or total cost of ownership.

The agent is not disobeying the objective. It is exposing the difference between the stated metric and the intended outcome. This is specification gaming: the policy satisfies the formal objective without producing the result the designer wanted. DeepMind has documented this problem extensively, including cases in which better optimization produced more effective exploitation of errors in the reward specification. Google DeepMind

A more capable agent does not necessarily solve objective misspecification. It may become better at exploiting it.

The problem becomes harder when the business objective is multidimensional. Operations must balance speed, quality, cost, compliance, resilience, customer value, and employee capacity. Converting these dimensions into a weighted reward,

does not make the trade-offs objective. The weights encode management preferences. Those preferences can change by customer, transaction, market condition, and risk level.

Linear scalarization can also fail to represent parts of a non-convex Pareto frontier. In practical terms, some valid trade-offs cannot be recovered simply by adjusting the weights of a single combined score. This remains an active area of multi-objective reinforcement-learning research. Multi-Objective Reinforcement Learning

There is no mathematically correct optimum until the organization defines its utility function, constraints, and risk tolerance. That is a management responsibility expressed as an engineering problem.

Correct Rewards Are Still Not Enough

Even a carefully designed reward function does not guarantee that the learned policy will pursue the intended goal outside the training environment.

A policy can learn a behavioral shortcut that is correlated with reward during training. When the environment changes, the capability may generalize while the learned goal does not. This is known as goal misgeneralization.

The distinction is important. Specification gaming means the reward was wrong or incomplete. Goal misgeneralization can occur even when the training reward correctly rewards the intended behavior. The policy may still learn an internal strategy that happens to produce the right results during training but fails under distribution shift. Google DeepMind

In business environments, distribution shift is normal rather than exceptional. Policies change, customers behave differently, products are introduced, teams develop workarounds, and external conditions alter the meaning of historical data. The environment is also reactive: employees and customers change their behavior in response to the agent.

This means the transition dynamics,

are neither fully known nor stationary. The same action can produce different outcomes depending on latent factors the system cannot observe.

No prompt can guarantee robustness against an unobserved change in the transition dynamics. The system needs monitoring, state validation, policy versioning, drift detection, and controlled fallback behavior.

The Harness Defines the Effective World

My work on agent engines changed how I think about the harness. It is usually described as a wrapper that gives the model tools, memory, and guardrails. Technically, it does something more important: it defines the effective decision environment.

The harness determines the observation function: which parts of the real state become visible to the policy. It defines the action space: what the policy is allowed to do. It controls state persistence, tool semantics, feedback, constraints, and termination conditions.

For most enterprise agents, this creates a partially observable Markov decision process. The real business state is sₜ, but the agent only receives an observation oₜ generated from that state. The agent must maintain an approximate belief about reality from incomplete and sometimes contradictory evidence.

The CRM is not the customer relationship. It is an observation of that relationship. A policy document is not the operating policy. It is an observation that may be outdated or overridden by an exception. A tool response is not necessarily ground truth. It is an output from another system with its own latency, permissions, and data-quality problems.

If two materially different business states produce the same observation, the agent cannot reliably choose different actions for them. This is not a reasoning failure. It is an identifiability failure. The information required to make the distinction does not exist inside the agent’s observable world.

No amount of prompt engineering can recover information that the environment does not expose.

Why More Context and More Tools Can Make It Worse

At the product layer, the standard response is to add context and tools. This sometimes improves capability, but it also expands the state and action spaces.

More context creates retrieval noise, stale evidence, conflicting instructions, and greater sensitivity to context ordering. More tools create overlapping affordances, larger search spaces, additional failure modes, and more opportunities for irreversible actions. External content also becomes part of the agent’s control surface, increasing exposure to prompt injection and data poisoning.

Memory introduces another layer of state drift. A stored memory may have been correct when written but false when retrieved. Unless memories have provenance, validity periods, conflict resolution, and explicit invalidation rules, they become ungoverned state.

This is where the harness becomes a prison. The agent can act only inside the representation created by the harness, but expanding that representation increases complexity without making it equivalent to the real organization.

Integrators usually attempt to manage this with more instructions, critics, and retries. These mechanisms change the distribution of trajectories, but they do not repair an incorrect reward surface, missing state variable, invalid transition model, or unidentifiable business condition.

They are treating an engine-level failure as a workflow-level defect.

Evaluators Do Not Eliminate the Problem

A common architecture uses one model to generate an action and another model to evaluate it. This can improve average performance, but it does not create an objective source of truth.

The policy and evaluator may share training data, representations, blind spots, and biases. Their errors can therefore be correlated. If both models find a result plausible, the system can confidently accept a wrong trajectory.

Learned reward models create the same vulnerability. The policy is optimized against the reward model rather than the human intention behind it. As optimization pressure increases, the policy searches regions where the reward model is poorly calibrated. This produces reward overoptimization and reward hacking.

Recent research has shown that reward-hacking behavior can generalize beyond the original task and that ordinary post-training mitigations may suppress visible behavior without removing the underlying tendency. Anthropic

For production systems, semantic evaluators should therefore be treated as probabilistic sensors, not final authorities. Hard constraints require deterministic validation, authoritative data, or human accountability.

The Architecture Must Separate Intelligence From Control

A reliable agent architecture should not ask the model to guarantee properties that probabilistic inference cannot guarantee.

The model can interpret unstructured requests, extract information, generate candidate plans, compare alternatives, and estimate ambiguity. Deterministic systems should enforce permissions, validate calculations, check mandatory conditions, manage transactions, preserve invariants, and block prohibited state transitions.

Critical workflow state should exist outside the model’s context in typed, inspectable fields. Actions should be idempotent where possible and reversible where necessary. Tool calls should have preconditions, postconditions, timeouts, and explicit error semantics. The system should distinguish between an action failing and an action returning no evidence of failure.

Constraints should not be encoded only as negative rewards. A sufficiently strong optimizer may accept a constraint violation if the expected reward is high enough. Non-negotiable requirements need constrained policies, action shielding, deterministic checks, or human approval.

Escalation is not a failure of autonomy. It is a control policy. The system should transfer authority when state uncertainty, model uncertainty, reversibility, or expected loss crosses a defined threshold.

The objective is not to maximize the percentage of decisions made by the agent. It is to maximize total operational value under explicit risk constraints.

Why COOs Must Own the Agent’s Objective

Most agent programs are still assigned to technical teams or integrators. That is a category error.

Engineers can implement a reward function. They cannot decide how the business should trade customer value against cost, speed against control, or autonomy against accountability. They also cannot infer undocumented operating knowledge merely by connecting more systems.

The COO must own the utility model, decision rights, constraints, escalation thresholds, and acceptable error distribution. Average accuracy is insufficient. A system with 99 percent accuracy may still be unusable if the remaining one percent is concentrated in high-value customers, regulatory decisions, or irreversible financial actions.

In the five engines I built, the recurring lesson was that upper-layer behavior is bounded by lower-layer definitions. If the objective is wrong, optimization makes the agent wrong more efficiently. If the state is incomplete, reasoning operates on a false world. If the reward is hackable, capability amplifies the exploit. If the evaluator shares the policy’s blind spots, the loop reinforces its own error.

These are not defects that an integrator can remove with a better prompt.

The Question After the Demo

The relevant question is not whether the agent can complete the task once. It is whether the full system defines the correct objective, exposes sufficient state, limits the action space, detects trajectory failure, and controls irreversible consequences.

A demonstration tests capability under selected conditions. Production tests the entire policy-environment-control system under uncertainty and distribution shift.

That is why many AI-agent projects fail even when the model appears intelligent. The failure began below the interface, before the integrator ever saw it.

The COO’s task is not to purchase more intelligence. It is to build an operating environment in which probabilistic intelligence can be used without confusing reward with value, observation with reality, or apparent convergence with control.