Government’s AI push must account for a new kind of social engineering

For years, the familiar phishing pattern was straightforward: Attacker → Human → Action. 

A convincing message persuaded someone to click a link, share credentials or transfer money. The defence focused on helping people recognise the deception and pause before acting. 

Artificial intelligence (AI) assistants introduce another step: Attacker → AI assistant → Human → Action.

An employee may ask an AI tool to summarise an inbox, review a document or identify urgent requests. The AI tool could produce a misleading summary or recommendation. 
 

Microsoft’s security guidance describes how malicious instructions embedded in emails can attempt to influence an AI assistant, while Google DeepMind has identified indirect prompt injection as a security challenge for AI systems that process external content.  

Consider a routine example: A supplier sends an email about a change to its payment details.

An employee asks an AI assistant to summarise the day’s correspondence. The assistant reports that the supplier’s payment instructions have changed and that the matter is urgent. 

The employee may give that summary more weight than the original email. After all, the organisation approved the AI tool and encouraged its use. 

This is where the risk changes.  

The attacker is no longer trying only to persuade an employee, but trying to influence the system that interprets information for that employee. 

AI changes the context in which people make decisions 

My work on phishing susceptibility uses a simple model: Personality + Emotion + Situation → Behaviour. 

People respond differently to the same request.  

A helpful colleague may feel compelled to assist. A conscientious officer may act quickly when a message appears to come from a superior. Someone under stress or time pressure may be less likely to verify an unusual instruction. 

AI can now shape the situation around that decision, by deciding which messages appear important, summarise what seems relevant and frame the next step.  

If its output is wrong or manipulated, it can add urgency and credibility to a request that should have been questioned. 

The concern is that an AI assistant may process untrusted content and present the result through a trusted channel.  

A malicious message can remain untrusted even after an AI has summarised it.  

A confident summary is not proof that a request is legitimate. 

Adoption needs to include security by design 

Governments are right to explore how AI can improve productivity, service delivery and access to information.  

But adoption cannot be measured only by time saved or tasks automated.  

Agencies also need to understand how AI changes the path from information to action. This means asking practical questions before connecting assistants to email, finance, HR, case management or procurement systems: 

• What can the assistant read, and what can it do? Access should be limited to what is needed. Reading a message should not automatically enable the system to forward data, change records or initiate a transaction. 

• How does it treat instructions found in external content? An email, webpage or document should be treated as information to assess, not as authority to override the user’s instructions or the system’s safeguards. 

• Which actions require independent verification? Changes to bank details, disclosure of sensitive information and other consequential actions should be confirmed through an established channel, separate from the message that prompted the action. 

• Can staff see the source behind a summary or recommendation? Employees need a way to inspect the underlying message and understand when AI output is based on external material. 

• Are systems being tested against manipulation? Security testing should include hidden or indirect instructions in emails and documents, alongside familiar phishing simulations. 

These are governance questions as much as technical ones.  

Procurement, security, records management and operational teams should consider them together.  

Staff training should also evolve: people need to learn that AI-generated advice inherits the uncertainty of the material it used. 

Trust must not be transferred automatically 

While AI assistants can make public servants faster and more effective, they can also make a false instruction feel clearer, more relevant and more credible. 

The central leadership challenge is to keep that added convenience from becoming unexamined authority. 

A useful principle is that AI-generated advice should never inherit more authority than its source. 

If an assistant summarises an unverified email, the result remains unverified.  

If it recommends an action based on a document from an unknown sender, the recommendation does not make that document trustworthy. 

As governments accelerate AI adoption, leaders should ask two questions together: What value can this tool create? And how could an attacker manipulate the tool our employees are learning to trust? 

The next phishing attack may try to influence the AI first—and let the organisation’s trust in that AI carry the message the rest of the way. 

-------------  

The author is a former military security officer and senior communications leader with over 30 years’ experience. He helps organisations strengthen their human firewall by transforming employees from the weakest link in cybersecurity to the first line of defence.

He has previously worked for the Singapore Armed Forces, Prime Minister’s Office, and A*STAR, leading crisis response teams, advising political office holders, and building communication strategies that work under pressure.