The solution will offer seamless, powerful search and analytics to users, enabling cell-level queries of any concept held in the data lake. High-value or high-usage data is pushed into user-friendly visualization tools such as Kibana. “Regardless of where data is held, the hybrid architecture is invisible to end users,” says Cunningham. “The service will be able to be picked up at any point, providing the same experience to users wherever the data is located.”

The Service’s architecture will also support enrichment of data for better insights. “With the smart-meter data, for example, even though there are hundreds of millions of rows, you only have a few variables,” says Cunningham. “To do interesting things like figuring out how much energy users are saving with efficient appliances, you have to add more variables. With AWS, we will be able to scale to accommodate these increasingly massive data sets, supporting deeper insights.” The system is highly elastic, giving the Service the flexibility to accommodate unpredictable workloads.

In addition, the Service wants to focus on delivering engaging and accessible data on par with the best, most innovative digital experiences. “Using AWS means we can deliver greater value for public investment than if we tried to build infrastructure on our own,” says Cunningham. “AWS has demonstrated its commitment to helping us innovate using disruptive technologies, with a clear pathway to core operational infrastructure.”