Abstract
Organisations increasingly want AI systems that can do more than generate outputs for humans to act on – they want AI systems that can get things done. In the workplace, these already exist in the form of AI agents that can create software, manage first-line customer relations, automate various elements of office workflows, and perform a growing range of roles. Agentic AI is likely to be one of the most consequential technologies of the next decade because it could make organisations – and nations – that successfully deploy it more efficient, productive and effective than those that do not.
But giving AI systems access, authority and trust within complex organisations brings significant risks and can lead to serious adverse incidents. In a widely reported case, for example, around 1,200 OpenAI agents intended to run in isolation during an internal test of their cyber-offensive capability – with normal safeguards relaxed – instead coordinated on an unsanctioned channel to breach containment, with approximately 700 going on to join a multi-day online intrusion of a machine learning platform called Hugging Face. They appeared to be trying to understand and defeat the benchmark’s automated scorer, in ways their operators had not intended.[footnote 1]
Many of the greatest risks relate to how agentic systems behave when deployed within sensitive and high-stakes organisational contexts. This includes failures that occur when agents pursue goals as instructed yet do not behave as intended.
This paper provides an applied taxonomy for understanding agentic AI behavioural failure modes. It also outlines practical principles for applied behavioural assurance: the capacity to understand, monitor and govern how agentic systems behave in deployment. Behavioural assurance can be an effective enabler of agentic AI – without it, systems may not be deployed due to security and safety concerns, or may be deployed in unsafe and insecure ways. Realising the potential of agentic AI may depend as much on building organisations capable of governing these systems as on building capable agents.
Addressing the challenge of behavioural assurance should be a high priority for the UK national security community in the years ahead, given the significant potential consequences of agentic AI deployment in a range of security contexts.