It's time to say goodbye to the open data portal

Look at some of the most transformative operating models used by leading digital organizations such as Wikipedia, Twitter, and TripAdvisor and what do you notice? Their value comes from sourcing data from outside their organization.

In light of this, many cities’ approach to open data can be seen to have a glaring shortfall: the flow of information is in just one direction.

Businesses, charities, and citizens can receive data from the government. But – with a handful of exceptions such as the Leeds Data Mill – there are few official, standardized, and automated mechanisms to provide data to the government. That is a huge missed opportunity. External sources of data could be of enormous value to city authorities.

Consider, for example, how electronic payments companies know where people come from (based on the card address) and exactly where and how they spend their money – data that could provide powerful insights into how citizens and tourists interact with a city.

Consider, too, how mobile phone operators and companies like Google and Apple collect data from handsets to provide their services – data that could be used to create real-time maps of urban traffic or pedestrian flows. And consider supermarket retailers who obsessively monitor and analyze data on their customers’ purchasing habits – data that could help reveal the economic health of a local area.

All these examples speak to a more important point: in any given city there are many different creators (or owners) and consumers of data. Just as businesses have information that could be valuable to local authorities, citizens may have information valuable to businesses or volunteer groups.

Businesses may have data that could be repurposed and used by other businesses. In short, city data lies in many places and could deliver value to different people. The problem is that there is currently no mechanism to efficiently connect those with data to those that need it.

Too often data remains in its silos, its potential unrealised. So how can cities unleash the power of city data for the benefit of everyone? I believe the answer is to replace the open data portal with a City Data Marketplace.

The City Data Marketplace

A City Data Marketplace (CDM) would be an online marketplace that connects organizations and individuals that have useful data with those that want it. Similar to how a site like TaskRabbit.com connects those offering their skills for hire with those who need jobs done, the City Data Marketplace would bring together creators and consumers of data.

The site would have two types of listing: Creators: Individuals and organizations with datasets could advertise them on the site, specifying the quality, format, range, and scope of the data, together with the price (if any) they would charge for providing them. (Local authorities and public sector bodies could continue to use the same marketplace to offer their open data for free.) Consumers: Those needing data – whether a local authority, a business, a charity, or an individual – could advertise their data needs. They would specify whether it was for private/internal use (“shared data”), or whether they wanted to release it as open data.

Creators could then see that request and publicly bid to provide the data, specifying how they would source it, its quality, and so on. This marketplace solution would offer at least eight major benefits:

1. Increase the availability of previously inaccessible datasets.

The marketplace could help unlock thousands of datasets that currently remain hidden within their organizations. Though some private sector organizations such as Google, Uber, and Vodafone are already offering some of their data to cities, creating a mechanism to monetize data would provide an incentive for many more businesses to share their data. For the first time, people and organizations outside the government would be able to see what data public authorities need.

2. Increase innovation.

Instead of cities specifying the exact data they think they need, they could instead put a request on the CDM outlining the problem they want to solve. A variety of different individuals and organizations could then offer data that solves that problem in different ways.

Imagine a city that wanted data on traffic flow to better manage congestion. City officials may start with the assumption that they need to work with a technology vendor to install an IoT network with embedded sensors on every street.

However, by posting the challenge on a CDM, they may find that mobile network operators can solve the same problem at a fraction of the cost by providing aggregated and anonymized data from phone signals. Or perhaps a volunteer group could offer to create an app for free that helps citizens report congestion in their area. By focusing on ends rather than means, the marketplace would reveal the available options.

3. Competitive pricing.

Some organizations could offer data for free (e.g. local authorities offering open data), and others might charge, but the amount requested by consumers or listed by creators would be publicly viewable, creating competition to find the best price for any particular type of data. As a result, data from different sources would be more accurately priced. (I’d suggest that pricing should based on the value of the outcomes the data can enable rather than on a per-byte basis.)

4. Review and feedback.

Similar to reviews of buyers and sellers on sites like eBay, the quality of datasets and data creators would be reviewed and ranked so that others could determine whether the data provided by each organisation was to the quality specified, encouraging better practice.

5. Potential new revenue stream for city authorities.

If city authorities hosted the marketplace, they could charge a small commission on any data sales conducted through the platform, helping to cover the costs of its creation and maintenance.

6. Help spread best practice.

Currently, there is no mechanism for local authorities or cities to see how other regions are using and paying for data to solve specific urban issues. Using the CDM, best practices could be shared as all purchases of data would be publicly viewable.

7. Highlight which open data sets cities should provide.

The UK currently seems to measure its success on open data by the number of datasets it lists online. That is a poor measure. What matters is what the data is used to achieve. By offering open data alongside paid-for data from other sources, it would be more transparent to cities about which data sets were of real value to their communities.

8. Policy based on hard data in place of modelling.

By providing access to a wealth of new data, cities would be able to design better initiatives and policies based on actual data rather than restricted estimates or models.

What about privacy?

The introduction of CDMs would need to be accompanied by proper protections for citizens’ data. The Information Commissioner’s Office (ICO) has a detailed Anonymisation Code of Practice.

Any organization wishing to sell or share their data via the marketplace should be obliged to certify and provide evidence that they have met that code. Where public sector organizations within cities wish to use private sector data, they would share responsibility for assuring that the data they procure has met ICO regulations.

From Open Data to City Data

The rise of the open data movement has already done much to change the way that local authorities and public sector bodies think about the value and purpose of the data they create, collect, and use. But it only represents the first step. For innovation to thrive, cities need to move from open data to city data – unleashing data from wherever it may reside and making it available to those who can best use it.

To that end, it’s time to upgrade the humble open data portal and turn it into the data marketplace that cities need. For further details and case studies on this idea, see Smart Devolution – a report I co-authored with Cameron Scott for the think tank Policy Exchange.

This article was written by Eddie Copeland, Innovation Lab Director at Nesta - the UK's innovation charity. Image used under a Creative Commons License