An organisation deploys AI agents into its customer operations and, sensibly, builds a monitoring framework around them: dashboards, accuracy thresholds, a review triggered when the numbers drift. Months in, the reviews keep arriving too late. By the time a threshold trips, decisions have already been made on degraded output. More monitoring is added. It keeps lagging. The instinct is that the tooling is not yet good enough. The real problem is older than AI, and no amount of better monitoring will reach it.
The current conversation about governing AI agents treats one thing as settled: the challenge is new. AI agents are different from traditional software because they are probabilistic rather than deterministic. Deterministic software runs identically every time — give it the same input and you get the same output. AI agents interpret context, make judgements, and produce different outputs in similar situations. That variation, the argument goes, is why old approaches to software management fail.
The observation is accurate. The conclusion is not.
AI agents do behave probabilistically. But the conclusion that this is what makes them hard to govern misses an older and more fundamental issue. Managing any controller operating in a complex environment is not a problem created by AI. It is a structural property of complex systems, and it has been understood since the 1950s.
The cyberneticist W. Ross Ashby described what he called the Law of Requisite Variety. The law states that a controller can only manage a system as complex as the controller itself. If the environment produces more variety — more unexpected states, more unpredictable combinations — than the controller can match in response, the controller will eventually fail. It does not matter how well-engineered the controller is. If it cannot match the variety of what it is governing, it will be overwhelmed.
This is not a law about AI. It is a law about control in complex environments. A hospital managing patient flow, a supply chain responding to disruption, a government department handling casework — all of them face the same structural requirement. The governing system must be capable of matching the complexity of what it governs. When it cannot, it fails. Not because it is badly built, but because its design assumed a simpler world than the one it is operating in.
Traditional software management treated this assumption as though it were safe. Write the code, test it, ship it, maintain it. The underlying premise was that environments were stable enough that a fixed controller would suffice. For many applications, this held reasonably well. For complex, adaptive environments — customer service, clinical triage, regulatory compliance — it was always a simplification. The environment was always producing variety the controller could not match. AI agents make this visible faster because they operate in domains where the environment is not stable, and their outputs reflect that instability in ways that conventional software did not.
Organisations now building monitoring frameworks and calibration loops for their AI deployments are not solving a new problem. They are belatedly solving an old one. And because they believe the problem was created by AI being probabilistic, they are building responses designed to address the symptom rather than the structure.
The symptom is that outputs vary unexpectedly. The structure is that the control architecture does not have sufficient variety to match the environment it is trying to govern. Addressing the symptom means watching outputs and adjusting them when they drift. Addressing the structure means asking whether the governance system as a whole can actually match the complexity of what it is trying to control, and building something different if it cannot.
This matters in practice. A monitoring system that tracks outputs on a scheduled cycle and flags a review when metrics fall below a threshold will always lag. By the time the threshold is crossed, many decisions may already have been made on degraded outputs. A governance architecture designed to match environmental variety would include continuous signals that bypass the reporting cycle — alerting when something is wrong now, not at the next scheduled review. These are different architectures, not different settings on the same one.
The frame that produced the Software Development Lifecycle assumed sufficiently stable environments. That assumption made sense for a certain class of software in a certain era. It also made it invisible that the control problem was structural rather than technical. AI agents operating in complex social environments have broken the assumption. Most organisations responding to the break are still inside the frame that produced it. They are adding better monitoring to an architecture that was always insufficient, rather than asking whether the architecture needs to change.
Addressing a symptom with tools built inside the frame that generated it is a recognisable pattern. It is what happens when a frame-failure is treated as a local fault rather than a structural one. The instinct is to fix the part that is visibly not working. The result is that the system continues, slightly patched, until the same failure reappears in a different place.
Related
- Frame failure — the underlying condition: when assumptions no longer fit the environment, and more effort inside the frame cannot resolve what is wrong.
- Why the frame cannot see itself — why organisations building monitoring frameworks cannot easily see that the monitoring architecture itself may be the problem.
- Openness is not omniscience — on the difference between systems that remain open to signals outside their current categories and systems that retreat to what they can already measure.