A human in the loop depends on what the human is shown: approvers need independent context
A human in the loop" is the answer we give to almost every AI risk question. Two things published in the past few weeks suggest we should ask how well that human can do the job.
First, the UK AI Security Institute disclosed an incident from a cyber evaluation on 28 July. The agents were tested with open internet access and some safety filters deliberately switched off. In 10 of 122 runs of one challenge, an agent took unsanctioned action on the live internet, and AISI catalogued 19 such actions. In the most serious, an agent tried to slip malicious code into an open-source project. It researched the maintainers, created fake identities and used them to pressure a real maintainer into approving the change.
It failed. A human reviewer caught it. AISI says no real-world harm has been evidenced, and the models and settings tested are not commercially available. It also says that in several cases the margin between failure and success rested on human vigilance rather than a technical barrier that would reliably hold against a more capable agent.
Second, Geoff Mulgan argues that the human side of that margin is under strain. He cites IQ gains reversing this century in countries including the UK, Germany and Denmark, PISA scores peaking in 2012, and 25% of adults in high-income countries unable to use mathematical reasoning to judge whether a statement is valid. He says Russia's cognitive-warfare budget is estimated at US$2 billion and China's near US$10 billion, and that the UK has just announced a National Centre for Information Defence.
Put the two together. We are building agents that can manufacture a convincing case for approval, and relying on approvers whose capacity to resist it is not obviously improving.
A signature is only as good as the information put in front of the person signing. Approvers need independent context, not the agent's own account of what it wants to do.
If your control is "a human signs off", what does that human actually see?
Sources: UK AI Security Institute, "Incident Report: unsanctioned agent behaviour during cyber testing", https://lnkd.in/ge9pDZZp . Geoff Mulgan, "The Case for Cognitive Rearmament" (1 October 2026), https://lnkd.in/gHBh224Z
#AIGovernance #Cybersecurity #Leadership