Rethinking control, risk, and trust in the public sector

This article is part of a Thoughtworks thought leadership series on unlocking public sector AI innovation. You can click the author's byline to read other articles by him.

As artificial intelligence (AI) systems become more capable, they are also becoming less predictable in how they behave.

This introduces a different kind of complexity: One that is less about scale and more about predictability and control.

The question is no longer limited to whether a system works. Increasingly, it extends to how and why it produces the outcomes it does, and whether those outcomes can be relied upon in a consistent and accountable way.

For public officers who answer to the citizens they serve, that distinction is one between a system that can be trusted with service delivery, and one that cannot.

When outcomes are correct, but behaviour is not

One of the more unusual characteristics of modern AI systems is that they can arrive at the correct outcome in ways that were not anticipated.

In one example, an AI agent tasked with completing a benchmark did not solve the problem directly.

Instead, it identified the structure of the benchmark itself and worked around it, producing the correct answers without performing the intended task.

This is not a traditional failure mode. It highlights a different kind of risk, where systems can meet defined criteria while diverging from the intent behind them.

In regulated or high-trust environments, which are the kind public agencies operate in every day, this raises important questions about how systems should be evaluated and validated.

Rethinking what "control" really means

Historically, control in software systems has been closely tied to predictability. If a system behaves as expected, it is considered to be under control.

With AI, that relationship becomes less straightforward. As systems become more adaptive, their internal decision-making processes can be harder to interpret using traditional methods.

This does not mean control is lost, but it does mean it needs to be redefined.

The control shifts toward a combination of observability, guardrails, and continuous monitoring, including the ability to trace decisions, detect anomalies early, and intervene when needed.

Rather than relying solely on predefined behaviour, organisations need to ensure they can see how systems are behaving, detect deviations early, and respond appropriately.

New risks beyond traditional models

The introduction of AI also brings new categories of risk that do not fit neatly within existing frameworks.

These include behavioural unpredictability, questions around cost as usage scales, and broader concerns about sustainability.

At the same time, data becomes an even more critical dependency. As systems rely on larger and more interconnected datasets, questions around access, ownership, and governance become more complex, and more important.

In environments where accountability and public trust are central, these risks cannot be treated as purely technical concerns.

Data: the mechanism for maintaining trust

As these systems evolve, data becomes a key mechanism for maintaining control, traceability, and accountability.

Approaches such as domain-oriented data ownership and data products are beginning to help organisations balance accessibility with governance.

By making data more structured and intentional, they create clearer boundaries around how it is used, while still enabling innovation.

This balance is essential in contexts where trust is closely tied to how data is handled, shared, and protected, which is the standard that public institutions are held to.

Leading without complete visibility

One of the more notable shifts is how leaders are responding to these challenges.

Rather than waiting for complete clarity, many are choosing to engage with these systems directly.

These include experimenting in controlled ways, learning from outcomes, and gradually building confidence in how they behave.

This approach requires accepting that not all variables can be fully understood upfront.

It also requires ensuring that experimentation happens within appropriate boundaries, so that learning does not come at the expense of accountability.

Trust as the defining constraint

There is no simple resolution to these challenges, and many of the questions raised by AI systems are still evolving.

What is becoming increasingly clear, however, is that trust will be the defining constraint.

It is not just whether systems are capable, but whether they can be relied upon in ways that align with expectations, responsibilities, and the broader environments in which they operate.

Navigating this landscape will require new approaches to control, new ways of understanding risk, and a continued focus on maintaining the trust that underpins every system, service, and outcome the public sector is responsible for delivering.

------------------------

James Lewis is a Thoughtworks Distinguished Engineer and Director based in the UK, internationally recognised for software architecture and its intersection with organisational design and lean product development. He co‑authored the original 2014 definition of microservices with Martin Fowler, helping catalyse the industry’s move to independently deployable services. He serves on the team behind the Thoughtworks Technology Radar and regularly advises executive teams on aligning technology strategy and organisational structures to accelerate delivery.