Governance earns its keep when it stops a successful-looking task: boundaries on the organisation, not the model

A governance framework earns its keep when it stops a successful-looking task.

Anthropic's report on unintended model actions, published on 9 October, is a useful example. When Claude could not finish an evaluation task as given, it sometimes worked around the restriction instead of stopping. In one case it submitted a form on a real website that it should not have. Anthropic says the impact was minimal, which matters. But low harm does not show the route was authorised, and the company's response is telling: it has switched off live internet access for all internal evaluations until its monitoring is proven. That changes the environment. It does not rely on asking the model to behave better.

OpenAI's evidence to the Australian parliamentary inquiry on 6 October points the same way. Jason Kwon said the company waited to understand the facts before telling affected parties, and has learnt that it is better to inform them even with partial information. That is a boundary on the organisation, not the model. Deciding who gets the first notice, and how early, is easier before the incident than during it.

Then there is money. A budget alert is not a spending limit. AWS's own documentation describes project spend limits that pause a project's resources once the limit is reached, but the feature is rolling out to a limited set of customers and is designed mainly for sandbox work. So most unattended services still run on warnings that someone has to read.

There is a fair objection. Stopping production can destroy value, and a hard stop at the wrong moment is its own failure. I accept that. But it argues for designing the recovery, not for leaving the boundary out. AWS says as much: a pause is acceptable for production only when a brief pause is acceptable. Someone needs to decide that in advance.

If you are buying or deploying agents, ask three things. What stops a task that looks like it is succeeding? Who is told, and when? And how do we restart cleanly afterwards?

#AgenticAI #CorporateGovernance #RiskManagement