When the AI Harness Becomes the AI Prison

Over the past few years, while developing software engineering agent harnesses at SaasChing AI, I started noticing a recurring pattern. Every improvement we made to the harness made the agents better at their jobs, but it also made them more dependent on the environment we had created for them.

This is not a new problem in AI. Reinforcement learning has dealt with versions of it for decades. Research on generalization has shown that RL agents can become highly specialized to their training environments rather than learning generalizable behavior. CoinRun demonstrated this explicitly by separating training and test environments and showing substantial generalization gaps. Under partial observability, the problem goes deeper: the observation available to an agent is not necessarily the true state of the environment. POMDPs formalize this distinction, and belief-state methods attempt to maintain a representation of the hidden state from the history of observations and actions. More recent model-based RL systems learn latent world models and use them for prediction and planning.

The same issues are now appearing in LLM agents, but with an interesting difference. Modern agents don’t just learn a policy inside an environment. They can inspect code, reason about systems, call tools, interpret failures, and potentially discover mechanisms that were never explicitly represented in their harness. SWE-agent has shown how strongly performance depends on the agent-computer interface itself, while ToolSandbox found that state dependencies and incomplete information remain difficult even for strong tool-using LLMs. Voyager demonstrated another part of the picture: an LLM agent can use environmental feedback, execution errors, exploration, and accumulated skills to adapt to new situations.

That brings me back to something I wrote about recently (The AI Harness Illusion: It’s Really About World Models): the distinction between intrinsic and extrinsic world models. In that article, I described the harness itself as an extrinsic world model. I would refine that slightly. The harness defines the interface through which the agent observes and acts on the external world, and therefore strongly shapes the extrinsic world model the agent constructs. But the two are not identical. The actual environment can contain state, actions, dependencies, and causal relationships that the harness does not expose at all.

That distinction turns out to be important. The harness becomes a prison when the agent mistakes the world exposed through the harness for the world itself.

The problem starts when the boundaries of the harness become the boundaries of the agent’s world model.

The Smoke-Test Agent

One place where this becomes visible is in the smoke-test agents we build at SaasChing AI.

These agents don’t operate only against source code. They test deployed systems spanning web applications, backend services, APIs, and databases. An action in one component can change state somewhere else, and the resulting effect may not appear where the original action was performed.

In one case, an agent needed to perform an operation for which we had not given it a purpose-built tool. Instead of treating the missing tool as the boundary of what it could do, it inspected the implementation, understood how the operation worked underneath the abstraction, and found another path through the code.

This is a small example, but it exposes an important distinction.

The tools exposed by a harness are not the environment.

They are an interface to the environment.

If the harness provides deploy(), queryDatabase(), runTests(), and getLogs(), it is tempting to treat those operations as the action space of the agent. But the underlying system has many more possible actions and relationships than the tool definitions describe.

The agent in this case managed to reason beneath the abstraction. The action it needed was absent from the harness, but the capability still existed in the underlying system.

This is exactly the behavior we should want to preserve as harnesses become more sophisticated.

The Environment Is Larger Than the Interface

There is a deeper problem, however. The harness doesn’t merely hide possible actions. It also hides much of what those actions actually do.

Consider a deployment.

From the harness, the interaction might look like:

But that is only the observable interface.

The deployment might restart services, execute migrations, invalidate caches, change routing, publish events, trigger webhooks, restart workers, or initiate asynchronous processes in other systems. Some effects are immediate. Others may appear minutes later. Some may never become directly observable through the deployment tool.

So success does not describe the new state of the world.

It describes one observation about it.

This is where the POMDP analogy becomes useful. In a partially observable environment, an observation is not equivalent to state. The agent has to maintain a belief about the latent state of the system based on its history of observations and actions. Modern POMDP research continues to develop learned belief representations precisely because conflating observations with state leads to poor decisions.

For software agents, this distinction becomes critical.

A deployment succeeds. Three minutes later, a database changes unexpectedly. Five minutes after that, a web application begins returning errors.

A conventional tool-driven agent may see three separate observations.

An agent with a sufficiently rich world model can ask whether they belong to the same causal chain.

Perhaps the deployment triggered a webhook. The webhook published an event. A worker consumed it. The worker changed database state. That state eventually caused the application failure.

None of that causal structure may have been represented in the harness.

The agent has to infer it.

The Old Reinforcement Learning Problem

This has a direct parallel with reinforcement learning.

An RL agent learns through interaction with an environment. Its behavior becomes optimized against the states, actions, transitions, and rewards available in that environment. The better it becomes at exploiting those regularities, the better its measured performance.

But high performance does not guarantee generalization.

CoinRun made this particularly clear. Agents that performed well on training levels could perform significantly worse on unseen levels. OpenAI described the underlying problem as agents latching onto specifics of their environments rather than learning generalizable skills. Increasing environmental diversity and stochasticity helped reduce that gap.

The lesson is relevant to agent harnesses.

Every stable property of a harness is something an agent can learn to depend on.

If failures always produce logs, logs become part of the policy. If tests are always available, test execution becomes part of the reasoning loop. If deployment status reliably represents system health, the agent can begin treating deployment status as evidence about a much larger state than it actually observes.

Optimization turns environmental regularities into assumptions.

The problem becomes more severe when those assumptions are hidden.

I Had Seen This Before

Years before working on software engineering agents, I encountered the physical version of this problem in autonomous driving at NVIDIA.

A driving system could perform extremely well within the operational domain represented by its training and evaluation environments and then degrade when enough characteristics of the environment changed. Road geometry, lane markings, signage, weather, sensor conditions, traffic behavior, and even the unwritten conventions between drivers could change across regions.

Some failures were simply unfamiliar objects or situations. Those could often be addressed with additional data.

The harder problem was recognizing that assumptions about the environment itself were no longer reliable.

There is a difference between encountering something you have never seen and discovering that the world does not work the way you assumed it did.

Software agents increasingly face the same distinction.

When the Harness Becomes the Prison

This is where I think the current approach to agent harnesses eventually reaches a limit.

We keep improving agents by expanding the harness. We add better repository search, better context management, more tools, more memory, richer execution environments, better observability, more structured feedback, and increasingly sophisticated recovery loops.

And it works.

But every improvement also makes the artificial environment more complete. Eventually the agent can become extraordinarily effective inside a world we have carefully constructed for it while having little reason to question whether that world accurately represents the system underneath.

That is when the harness becomes a prison.

The problem is not that the harness constrains the agent. Constraints and abstractions are necessary. The problem is that the agent mistakes those abstractions for the boundaries of reality.

The available tools become the perceived action space.

The available observations become the perceived state.

The encoded workflows become the perceived causal structure.

The agent can reason extremely well within those assumptions while being unable to reason about the assumptions themselves.

Belief Instead of State

POMDP research offers one useful way of thinking about the solution.

Instead of assuming that an observation describes the current state, the agent maintains a belief state: a representation of what the underlying state might be given everything it has observed so far.

This changes the meaning of tool responses.

is no longer interpreted as:

The deployment is complete and the system is healthy.

It becomes evidence:

The deployment system reports success. Given what I know about this system, what does that imply about its current state, and what remains uncertain?

That uncertainty matters.

Recent work continues to make belief-state representations more practical for high-dimensional partially observable environments, although POMDP planning remains computationally difficult in the general case.

For LLM agents, we may not need to reproduce classical POMDP machinery literally. But the conceptual separation is important.

Observation is not state.

And I would add another:

Tool definition is not causality.

From Tool Use to World-Model Adaptation

Most current agent loops are organized around action selection:

When something fails, the agent replans and selects another action.

A more general agent needs another level:

The difference appears when reality violates prediction.

Suppose an agent deploys a service and predicts that A and B should change. Instead, C changes.

The normal response is to deal with C.

A world-model-driven agent should also ask why C changed.

Perhaps its causal model is incomplete. Perhaps an unknown webhook exists. Perhaps another service subscribes to an event. Perhaps there is a database trigger, queue consumer, background process, cache, or external integration that the agent does not yet know about.

The unexplained observation becomes evidence of missing world structure.

This is where LLM agents may have an advantage over traditional RL policies. An LLM agent can potentially search code, inspect configuration, query infrastructure, read documentation, trace calls, construct hypotheses, and perform experiments specifically to discover the missing relationship.

The objective is no longer merely to choose the best action.

It is sometimes to learn what world you are acting in.

The Harness as Sensor and Actuator

This leads to a different way of thinking about agent harness architecture.

The harness should not be the world model.

It should be the agent’s sensor and actuator interface to a larger world.

Tools provide ways to act on that world. Logs, APIs, databases, browsers, traces, and other systems provide partial observations of it. Neither should be assumed complete.

Above those interfaces, the agent needs a revisable model of what it believes exists and how those things interact.

That model might contain known components, causal relationships, expected effects, uncertainty, unexplained observations, and hypotheses about hidden mechanisms. More importantly, the agent needs mechanisms for changing that model when observations stop matching predictions.

Model-based reinforcement learning has been pursuing related ideas for years by learning latent dynamics that can be used for prediction and planning. POMDP research adds explicit reasoning about hidden state and uncertainty. LLM agents add another capability: they can potentially reason symbolically about the structure they are discovering and use existing software artifacts as evidence.

These threads are beginning to converge, but the problem is far from solved. Partial observability remains difficult even in modern RL research, and current LLM tool-use benchmarks still show significant difficulty with state dependencies and insufficient information.

Escaping the Prison

I don’t think the answer is a smaller harness.

Better harnesses will continue producing better agents. We should give agents good abstractions, reliable tools, useful memory, strong observability, and effective feedback.

But we should also make it possible for them to distrust those abstractions.

An agent should know that its tools expose only part of the available action space, its observations expose only part of the state, and its current understanding of causality is provisional.

When reality behaves differently from prediction, it should be able to distinguish between three possibilities: the action was wrong, the observation is incomplete, or the world model itself is wrong.

Those require very different responses.

The problem is as old as reinforcement learning: an agent learns the world we give it.

What is different now is that an LLM agent may be capable of realizing that we gave it the wrong world.

The next generation of harnesses should be designed to let it find the larger one.