If your AI risk register says ‘human review', send it back

A Florida lawyer told an appeal court that he had found the fabricated authorities in an artificial intelligence (AI)-generated draft, removed them and prepared a corrected version.  

The uncorrected draft was filed anyway. 

That is what makes JMOR Properties v Artist Alley Townhomes more interesting than another case about AI hallucinating law.  

The workflow included human checking, yet the wrong document still reached the court. 

The usual response is that people need to check AI more carefully. 

I think that misses the real control failure. “Human review” tells us that somebody was involved.  

It does not tell us what they could see, what they were expected to verify, or whether their approval remained attached to the thing that was ultimately used. 

My background is in operational risk and investigations. 

If somebody told me a serious risk was controlled because “a supervisor has checked it”, I would send the assessment back. 

I would want to know what they were checking, what failure they were meant to spot, what information they had available, how they would know something was genuine and what happened after they approved it. 

Those are ordinary assurance questions. Yet with AI they are often compressed into two reassuring words. 

Devil is in the details 

In Elliott v New York Bariatric Group, a self-represented litigant hid tiny white text on a white background inside court filings. 

Most of it was aimed at any AI system that might read the document, instructing it to agree with his position. 

The Connecticut court was not using AI to decide the filing, so the attempt went nowhere.  

But it exposed the vulnerability all the same: the human reader and the machine can be given the same file and still receive different information. 

A person can open a document and see one thing, while software parsing the same file receives something else.  

If information capable of influencing the machine sits outside the reviewer’s view, calling a human “in the loop” does not fix very much.  

They cannot supervise something they cannot see. 

Arbuckle v Tanner raises a different problem. AI-generated court transcripts appeared to carry a genuine court transcriber’s name, registration number and signature, even though she had not transcribed or certified them.  

Opposing counsel obtained an independent transcript, compared it with what had been filed and found 39 discrepancies. They then contacted the transcriber whose details appeared on the documents. 

What interests me more than the bad transcript itself is how the problem was found. Nobody spotted it by staring harder at the same document. 

They went outside it, found another source, compared the two and checked the certification with the person whose identity was being relied upon. That is much closer to what meaningful verification actually requires.

Nobody can rebuild every document from first principles.  

When signals of trust can be faked 

We rely on signals of trust: a familiar format, a recognised name, an official reference, a signature or a certification. Public bodies do the same every day.  

AI complicates that when something can look authoritative without being authoritative. The reviewer may now have to check not only whether the content is right, but whether the things making it look right are genuine. 

The JMOR case also exposes a much more mundane problem.  

Even if somebody spots an AI error and corrects it, what ensures that the corrected version is the one actually filed, published, sent or acted upon?  

It sounds mundane beside hallucinated cases and hidden prompts, but it is the sort of problem that breaks real processes. A good review provides no protection if another version leaves the organisation afterwards. 

This is why I think “human review” is doing far too much work in AI governance.  

It can mean at least three quite different things: making sure the reviewer can see what matters, giving them a proper way to verify what they are being asked to trust, and making sure their approval remains attached to whatever eventually gets used.  

A single line on a risk register tells me none of that. 

UK government guidance already recognises that meaningful human oversight depends on reviewers having enough expertise, time and authority.  

I would take that one step further.  

What human review in AI really means  

If human review is being claimed as a mitigation, the organisation should be able to explain exactly what the reviewer is controlling and what evidence shows that the control works:

1. I would want to know whether reviewers can actually catch realistic failures under the conditions in which they work.  

2. I would want approval tied to a specific version, so a later change is visible rather than quietly replacing what was checked.  

3. I would also want reviewers to know when machine-readable material can influence the system without appearing in the ordinary document in front of them. 

Importantly, I would want to know how much work all of this creates. 

That part is easily lost when organisations talk about AI productivity.  

If a tool saves ten minutes drafting something but an experienced member of staff then spends fifteen minutes checking sources, validating references, correcting errors and making sure the right version is released, the work has not disappeared.  

It has moved. That may still be a perfectly sensible trade, but it should be visible. More importantly, the organisation needs to know who is carrying that verification burden. 

If the safe use of an apparently efficient tool depends on a small number of experienced staff being able to recognise false authority, check provenance and keep control of the right version, that expertise is itself part of the safeguard.  

Increase the volume, lose some of that experience or spread those people across too many AI-enabled processes and the control changes, even though the risk register may still contain exactly the same words: “human review”.

None of this is an argument against people remaining involved in consequential AI use. I would be more concerned if they were not. It is an argument against pretending that “human review” is one control. 

A person can be involved and still be unable to see the information that matters.  

They can see it and have no reliable way of knowing whether it is genuine. They can find the error, correct it and still have the wrong version leave the organisation.  

Those are different problems, and they need different answers. 

So if I saw “human review” sitting on its own in the mitigation column of an AI risk assessment, I would send it back.  

Not because the human does not matter, but because I still would not know what that person was actually controlling. 

The job title is not the safeguard. What the person can actually see, check and stop is.

The author writes in a personal capacity and his writing does not reflect the views of his employer or GovInsider.

--------------- 

The author is a UK public sector practitioner with more than 17 years’ experience across policing, investigations, safeguarding, operational leadership and risk. He holds a First-Class BSc (Hons) in Applied Criminal Justice and is studying an MA in International Relations, Security and Strategy. He writes on AI governance, organisational learning and the integrity of public-sector decision-making.