Building resilience against evolving AI risks within the UK government

Artificial intelligence (AI) systems are evolving faster than the governance structures designed to oversee them.

Peter Wallich, who previously helped prepare the UK government for advanced AI risks as the senior AI risk manager at the AI Security Institute (UK AISI), says to GovInsider that red-teaming exercises have revealed that AI models can bypass safeguards “in ways institutions were not fully expecting.”

Now a senior research programme manager at the Constellation Institute, a US-based non-profit focusing on AI safety, he develops programmes and mentors frontier AI safety researchers in the UK and the US.

He says that the UK AISI, a state-backed research organisation under UK's Department for Science, Innovation and Technology (DSIT), was set up to address gaps in frontier AI safety techniques and understanding.

While it has been granted early access to frontier models before they go public, not all risks can be caught at this stage. Some only surface once systems are actively in use across companies and government agencies.

As a result, structural problems remain as evaluation cycles tend to lag behind deployment.

Since the UK AISI’s rebranding from “safety” to “security” last year, Wallich explains that AI safety has moved beyond solely technical issues and become a national security issue, with more AI security institutes established compared to three years ago when it started.

The shift and readiness to cope with evolving AI safety

The new name reflects the organisation’s strategic focus on larger-scale risks, including cybersecurity, as well as strengthening resilience against them.

Wallich highlights that small gaps don’t look like a problem until they stack. As AI becomes more embedded in the processes, the accumulation of risks could result in a gradual loss of visibility and control.

This is what Wallich and his colleagues have described as an unpaid “AI safety debt”, referring to gaps between an AI system’s actual safety approach and the approach it needs.

Additionally, when an AI system has access to sensitive data, exposure to untrusted inputs and the ability to perform actions across systems can further lead to data breach, identity theft and eroded public trust.

Security researchers refer to this as the “lethal trifecta”, where a failure in one area can
quickly cascade into others. The risk increases when systems are deliberately manipulated.

As Wallich explains “jailbreaks”, he highlighted that “even with safeguards, systems can still be manipulated into behaving outside their intended boundaries.”

Oftentimes, such unintended access or actions are unlikely to be detected in real time.

Public sector’s current AI safety readiness

Governments are building readiness through talent development, including training programmes for civil servants to build foundational understanding of frontier AI capabilities.

For example, Wallich mentions the high demand for training he developed with a colleague at the UK AISI that has reached over 400 civil servants.

In his role at Constellation Institute, he manages the relaunch of the Astra Fellowship, an AI safety research accelerator that eases hiring bottlenecks and reduces uncertainty in accessing specialist AI safety talent.

His work responds to persistent capability gaps in how quickly AI systems are advancing and how public and private sectors can build the expertise to cope with them.

But Wallich highlights that AI safety headcount and investment at public institutions do not scale at the same pace as AI research and development in the private sector.

“For every one person working on preventing catastrophic AI risk, there are many more working to advance frontier capabilities.”

This influences what governments are structurally able to see and respond to.

How the UK AISI has dealt with public sector challenges

Public sector compensation and internal governance are often overlooked, says Wallich.

For example, in the UK, rigid promotion structures can lead to frustration, while lengthy and restrictive security clearance processes can prevent experts without a long history of UK residence from sharing their technical knowledge in classified settings.

Even well-intentioned practices such as name-blind recruitment or standardised panel assessments can unintentionally weaken the ability to identify niche technical talent, he notes.

Despite these constraints, the UK AISI has demonstrated that mission-driven work can still attract strong technical expertise through specialist recruitment flexibilities, including piloting “Class Approvals” that allow quicker hiring while still adhering to interview and oversight standards.

This aligns with what was discussed in the AI Safety Summit, where governments can only assess and govern frontier AI effectively if they are able to build and retain deep technical capability internally.

However, this success requires more effort to scale across the wider sector.

A matter of understanding AI’s impact and ways to intervene

AI is already shaping decisions across public services, from healthcare and welfare to taxation and digital government systems, often before anyone explicitly “approves” its role in the process.

As such, AI adoption has become a matter of understanding how decisions are made, and what you can do when the system starts behaving in unintended ways.

Ultimately, readiness depends on whether governments can navigate the evolving system embedded within critical infrastructure.

“We need to scale both the field and government capacity faster, because the gap between capability and oversight is still widening,” says Wallich.

“This is not just a question about existing or new regulation, but also the ability to interpret system behaviour and outputs and collaborate across technical and policy domains.”

Without this shift, accountability risks being an afterthought, leaving you to justify decisions you do not fully understand and cannot fully trace.

The views expressed in this article are those of the interviewee and do not reflect the official views of the UK AI Security Institute and Constellation Institute.