Accountability needs to be embedded in AI programmes

Governments are rushing to deploy citizen-centric services powered by Artificial Intelligence (AI), particularly, generative AI (GenAI) chatbots.

This has led to a scramble among public sector agencies to develop frameworks that ensure the ethical use of AI in public sector service delivery and strategy planning.

According to Splunk’s Group Vice President of Strategic Advisory for Asia Pacific, Robert Pizzari, accountability needs to be embedded across the different phases of the lifecycle of an AI programme, whether it is in deciding what models are being used, or how the data pipelines are being integrated into the actual sources of data.

“Another area of concern is tied to transparency,” Pizzari said.

It has been observed, particularly in the early use cases of GenAI, that there has been a tendency for some models, based on how they've been trained, to show a degree of bias towards different cultures and characteristics, he noted.

This is why, Pizzari said, training, transparency and the data source are important, especially in the public sector and “stakeholders, together with citizens and others, need to be made aware of that baseline.”

To subscribe to the GovInsider bulletin click here.

Aligned with that is the data privacy element, he added.

Developing GenAI

Pizzari observed that for GenAI, the starting point in developing trustworthy models has to be the data sets, and that “we need to look at what data will be used to train the models.”

Several factors need to be taken into consideration. “Things like, is it [training] data that you own, that you've curated, that has a single source of truth, meaning that it hasn't been tampered with, it hasn't been intercepted by a third party that injected data that will skew results,” he explained.

There is a security element to this, but the foundation is to ensure that we minimise the impact of bias which depends on good data governance, added Pizzari.

He noted that depending on the type of models being deployed and for what purpose, it will not always be data that the user agency has full control over.

“For example, let’s say Singapore is looking to change its immigration policy or look at how the population will change over time, and where will the economy benefit. For that, the government may require data sources outside the national data repository.

“Therefore, if you go to that next point in that data ecosystem, whom do you trust? The question is also how to verify and then be able to identify if there are any anomalies in that data set,” Pizzari said.

This all boiled down to what checks and balances one can put in place, he added.

“There is a continuous loop around building the pipeline, the sources of data, the authenticity of that data, and then being able to validate at each step, so that if biases are detected, let's say in results, the data team, together with the folks that are leveraging these tools and systems, can have that feedback looped back into the data, back into the pipeline, and the process to say, ‘Hey, we've got some issues here’”, he said.

To subscribe to the GovInsider bulletin click here.

“The need is to have a data governance framework and a zero-trust policy for going back to validate (the data) periodically or even in a structured format,” Pizzari added.

Trustworthy AI principles

Pizzari shared that  Splunk, with its trustworthy principles for digital resilience in AI, is building AI capabilities across two dimensions.

“One is foundational AI, and that is a collection of capabilities, some of which have been around for decades, and that is in the category of machine learning (ML); often we use the words AI and ML interchangeably”, he said.

Pizzari added that this foundational capability is something where Splunk’s expertise lies — supporting customers and enabling organisations to make sense of their data using GenAI.

He noted that GenAI has enabled easy access, and one can now query the programme in the native language about a data set, allowing the user to zoom into specific areas of interest to extract responses.

“This provides ease of use and accessibility, but we must ensure that we've built accountability around all our systems that will embrace human oversight. In other words, we must have a human-in-the-loop to make that final determination and decision,” he said.

The second major area is about transparency, Pizzari added.

It is necessary to be able to explain, with a high degree of accuracy and low error rates any result provided by the AI programme. If we cannot do that there is a loss of confidence and integrity in the service offering, he added.

To subscribe to the GovInsider bulletin click here.

The Splunk official added that the third pillar was really about privacy. “Today if you query or upload data into an open GenAI model, that data is forever part of the data set, and anyone across the globe can query that data.”

Policy around privacy

Having a policy around privacy and [AI] models is important, and that's what we offer to our customers, Pizzari said. “I think the other piece that ties back to the ability to be fair, is to run it in a fair, unbiased, neutral mode – this is quite important to avoid things like discrimination.”

The other key principle that Splunk is building into its models is tied to domain-specific applications.

“In our case, there are two principal domains, one for cybersecurity practitioners within their environments, helping them unlock insights and providing some prescriptive next steps to help with investigations to unlock areas that may have been overlooked.

“In other words, ask, ‘What am I missing? What else could I try given this context? What's the next step?’

“The other domain in which we are applying these principles is observability. So, the first is the human in the loop, the second is tied to the domain-specific areas.

“And then combining that with the areas of privacy and fairness, that's the general direction we're heading,” Pizzari said.

To subscribe to the GovInsider bulletin click here.

He added that the overarching North Star for Splunk and governments is tied to driving resiliency that ensures the safety, security, and resiliency of systems, whether in the face of downtimes in general or attacks.

The last piece that ties everything together is the ethical use of AI. “Without ethics, we cannot achieve anything,” Pizzari pointed out.

He noted many similarities between Splunk’s trustworthy AI principles and Singapore’s approach to AI governance.

Without a doubt, if we overlay what has been published by the Singapore government and Splunk’s approach on the broader trustworthy AI perspective, there is a lot of convergence around the emphasis on transparency, fairness and ensuring that there's a human-centric design element.

Like Singapore, Splunk also believes that these systems need to be trustworthy and provide a framework to guide development usage. “This is a Splunk mantra that's been embraced by the industry as well as the government”, Pizzari said.