Building whole-of-government observability capabilities on Elastic

With IT infrastructure resilience being a key focus area of Singapore’s updated Smart Nation policy, observability of websites, services, and applications has become an even more important tool to help government organisations proactively detect issues and quickly resolve infrastructure or performance-related incidents.

At a presentation, GovTech Singapore’s Lead DevOps Engineer, Yap Yi Sheng, noted that the agency’s StackOps team required a common observability tool that allows government agencies to

collaborate on monitoring data to improve system/site resiliency and quality via an IM-compliant observability platform - which collects and analyses logs, metrics and traces.

This would be used as part of the SG Tech Stack, which streamlines and simplifies the development process and enables code reuse across the Singapore government to build secure, high-quality applications.

He added that currently, there are many dependencies between different government IT systems, like platforms and tools that facilitate government e-services. Yap’s team wanted to develop a tool that could support data sharing in a secure environment.

Yap was speaking alongside Elastic Principal Solution Architect Lim Jie-Hong at a presentation titled Strengthening Public Services with AI: Elastic’s Strategies for Resilience, Security, Observability, and Citizen-Centric Operations at the AWS Public Sector Day 2024 event in Singapore on October 3.

To subscribe to the GovInsider bulletin click here.

Easy integration through public cloud services

Elastic’s strategic collaboration with Amazon Web Services (AWS) enables public sector customers using AWS to easily use the Elastic Search AI platform.

As of last year from GovInsider, AWS is hosting over 600 government digital services in Singapore.

Yap pointed to three reasons that GovTech was utilising Elastic Cloud on AWS for the StackOps component of the SG Tech Stack.

Firstly, Elastic provided improved observability through structured telemetry data (Elastic Common Schema) and contributed to OpenTelemetry, allowing open-standard, data enhancement of the structure, organisation, and relationships of monitoring data within a whole-of-government context.

The second reason, Yap said, was because Elastic was able to integrate its observability solution with the unique verification and authentication methods used by the government, including TechPass, SEED and GSIB.

Thirdly, Elastic’s solution was easy to use. It taps into automation or Infrastructure as code (IaC) to simplify deployment, enables platform analysis and sharing, provides observability maturity indicators to assess performance, and more.

Aside from the Singapore government, the UK’s meteorological office (Met Office) uses Elastic to scale up its capability to ingest data and drive better insights.

The Met Office’s data comes from many different sources, including supercomputers predicting weather patterns, Raspberry Pi devices, and Internet-of-Things (IoT) sensors, Elastic’s Lim added.

As noted in the online case study, the UK Met Office also runs a mix of on-premises, cloud and software-as-a-service platforms, and  Elastic is used to collect data from these various platforms.

To subscribe to the GovInsider bulletin click here.

AI to accelerate translation of data to outcomes

As artificial intelligence (AI) adoption accelerates, Lim also noted that the Elastic AI Assistant is not just another chatbot or chat interface embedded in the platform. The Assistant is built on the Elasticsearch Relevance Engine (ESRE) and powered by generative AI for smarter and more interactive search, security, and observability experiences.

Data is key when it comes to the validity of AI-generated insights. This is why Elastic’s  vector database offers users an efficient way to strengthen the accuracy and reliability of AI Assistant responses, he noted.

Unlike traditional databases, vector databases enable storage and access to the vector representation of both structured and unstructured data. This is important in deploying effective AI models, as the variety of datasets allows the model to hone in on accuracy, speed and scalability.

“When we combine these datasets together, they can provide users with a more contextualised response to their intent.

For example, if the user is troubleshooting a service outage and happens to be on a certain application window, AI can trace some of the telltale signs of what is going on from other applications’ performance”, Lim explained.

Lim shared the case study of Cisco having deployed Elastic for the former’s enterprise search platform that was able to integrate with an external cloud service provider.

According to the online case study, AI powers the auto-suggestions on the search created from passages on Cisco.com pages. The deep question-and-answer solution also enables the capability for hybrid search – which consists of semantic search, which interprets the meaning of words, not just keywords, and text search – on the passages.

As a result of better search relevance, it was able to save 5,000 support engineer hours per month, reduce the time spent on search queries by up to 73 per cent, and solved 90 per cent of support requests with its new platform.

To further strengthen its hybrid search capability, Elastic has also launched ELSER, a natural language processing (NLP) model, that enables users to search by their intent or context, rather than by exact technical terms.

Elastic will be hosting the ElasticON Tour Singapore on March 4 next year for interested parties wanting to find out how its AI-powered search solutions can supercharge app development, security analytics, log analytics and more.