Fund agent controls before rollout: the dead-tree rule

Jared Diamond spent fifty years in New Guinea and came home with one rule. His hosts refused to camp under a dead tree that he thought would stand for years. The tree was solid, but it was dead, and they would not sleep under it. Any single night was probably safe, yet they were not betting on one night. They were betting on the hundreds of nights they would spend in the forest over a lifetime.

Diamond later showed how the same logic works in ordinary life. A one in a thousand chance of slipping in the shower sounds like nothing, until you take 5,475 showers, which is about fifteen years of daily washing. At that rate, you should expect about five falls.

Now count the showers in your organisation. Every agent action, drafted brief and automated approval is one. Take a one in 10,000 chance of a bad outcome per action. Across 10,000 actions a year, the chance of at least one bad outcome is roughly 63 per cent, if the actions are independent.

They are not. Agents on the same model, with the same prompts and the same data, tend to fail together. You get fewer incidents and bigger ones, and the first arrives without warning because nothing in the monthly numbers looked wrong.

Arvind Narayanan and Sayash Kapoor recently argued that we underinvest in resilience against catastrophic and systemic risk, cyber most of all, and that the extinction framing crowds out the defences that would help. I agree. For an Australian agency or bank, the risk that matters is the dull one: a shared flaw repeated across thousands of low-stakes decisions that nobody reviews.

Insurers ask the right question about tail risk, which is whether it can be priced. If it cannot, the control belongs in the budget: logging, monitoring, and a named person who owns each decision type. Fund these before the next rollout, not after the first incident.

Which of your agents has an owner and a logged failure rate today?

#AI #Governance #PublicSector

https://lnkd.in/giiB4DCX