UK open contracting: good policy & maturing platform – it’s time to invest in implementation

[Summary: relecting on national open contracting progress in the UK]

Last week the Prime Minister issued a letter reminding central government departments of their transparency responsibilities and providing updated guidance on the information that should be disclosed and how. Amongst the guidance, is a revised note on “Publication of Central Government Tenders and Contracts” which provides a good snapshot of the current position for national government contracting (and which is also framed as useful guidance for Local Authorities considering their responsibilities under the local government transparency code).

The note covers:

  • The legislative requirement to publish most opportunities and awards over £10,000 via the Contracts Finder platform;
  • The policy committment of central government to see all tender documents, and contract texts attached to those notices on Contracts Finder;
  • Guidance on all the documents that go to make up the contract (and so that should be attached to Contracts Finder)
  • Re-iteration of the limitations to redaction of contract documents;
  • Recommendations on transparency clauses to include in new contracts, to have clear agreement with suppliers over information that will be public.

As contracting transparency policy goes: this is good stuff. We’re not yet at the stage in the UK of having the kind of integrated public financial management systems that give us transparency from planning to final payment, nor are their the kind of lock-in measures such as checking a contract has been published before any invoices against it are paid. But it does provide a clear foundation to build on.

The platform that backs up this policy, Contracts Finder, has also seen some good progress recently. With hundreds of tender and award notices posted every week, it continues to provide good structured data in the Open Contracting Data Standard through an open API. In the last few weeks, the data has also started to capture company registration numbers for suppliers – a really important key to linking up contracting and company ownership information, and to better understanding patterns of public sector contracting. The steady progress of Contracts Finder as a national platform (with a number of features also now added to help capture sub-contracting processes too) makes it absolutely key to monitoring and improving implementation of the policies described above.

There are still some challenges for the platform: data quality (and document availability) for many of the records in Contracts Finder relies upon the features of e-Procurement systems used by departments or local authorities to manage their contracting processes. If these systems don’t encourage inclusion of company identifiers, or contracting documents, we may struggle to reach full policy compliance and the best data quality. Ongoing improvements to the APIs for data entry, and to the tools for monitoring data quality, could certainly help here, as would increased engagement with e-procurement system vendors to get them to bake open contracting into their platforms, as Chris Smith has called for.

However, as we head in 2018, whilst we have to keep working on policy and platforms – the real focus needs to be on implementation: monitoring and motivating each department or public agency to be sure they are not only seeing transparency in procurement as a tick-box compliance excercise, but instead making sure it is embraced as a core part of accountable and open government. To date, Open Contracting in the UK has been the work of a relatively small network of dedicated officials, activists and entrepreneurs. If the vibe at OC Global last month was anything to go by, 2018 may well be the year it moves into the mainstream.

Disclosure/notes

I’m a member of the UK Open Contracting Steering Group, working under Commitment 5 of the UK OGP plan and I work for Open Data Services Co-op as one of the Open Contracting Data Standard helpdesk team.

On the journey: five reflections from #ocglobal17 (Open Contracting Global)

At it’s heart, open contracting is a simple idea: whenever public money and resources are at stake through a contracting processes, transparency and participation should be the norm.

Yet, as the Open Contracting Global Summit (#ocglobal17) in Amsterdam this week has demonstrated, it’s also an idea that brings together a very wide community. Reflecting on conversations from the week, I’ve tried here to capture five key reflections on where we are at, and where we might be heading:

(1) It’s not just procurement

Although the open contracting emphasis is often on the way governments buy goods and services, there are many other contracts where public resources are at stake: from licenses and concessions, to Public Private Partnership deals and grant agreements.

These each have different dynamics, and different approaches might be needed to open up each kind of process.

The Open Contracting Data Standard (OCDS) is primarily designed around procurement processes, although at OCGlobal we gave the first public preview of the OCDS for PPPs profile, that extends the OCDS data model to provide a structured way of recording in-depth disclosures for Public Private Partnership deals.

(2) It’s not just JSON

Thanks to Article 19, the corridoors at OCGlobal had been turned into a ‘gallery of redaction’. Copies of contracting documents obtained through FOI requests provided tantalising hints of government and private sector deals: yet with all the key facts blacked out. These stood as a reminder of how many times the public are kept in the dark over contracts

Neither documents, nor data, on their own will answer all the question citizens or companies might have about contracting. Not will they automatically spark the kinds of participation, scrutiny and engagement that are the essential complement of transparency.

Although publication of standardised data might be the most concrete manifestation of open contracting, it’s problematic to conflate transparency or open contracting with use of the OCDS JSON schema. Indeed, the 5-star model published as part of the guidance for OCDS 1.0 highlights that governments can taken their first steps towards open contracting data by publishing any contracting information on the web, stepping up to machine-readability and standarised data as capacity allows.

Any other approach risks making the perfect into the enemy of the good: preventing publication until data is perfect.

The challenge ahead is in designing and refining the incentive structures that make sure open contracting efforts do not stop at getting a few documents online, or some fields in a JSON dataset – but instead that over time they broadens and deepen both disclosure, and effective use of the information that has been made available.

(3) It’s an iterative journey

There’s a much refreshed implementation section on the Open Contracting website, curating a range of guidance and tools to help put open contracting ideas into practice. The framing of a linear ‘seven steps’ journey towards open contracting is replaced with a ‘hopscotch’ presentation of the steps involved: with interlocking cycles of development and use.

This feels much closer to the reality I’ve experienced supporting open contracting implementations, which involve a dance back and forward between a vision for disclosure, and the reality of getting data and documents published from legacy systems, transparency features added to systems that are in development, or policies and practice changed to support greater citizen engagement in the contracting process.

There was a lot of talk at OC Global about e-procurement systems as the ideal source of open contracting data: yet for many countries, effective e-procurement deployments are a long way off, and so it’s important to keep in mind different ways tools like OCDS can be used:

  • Based-on – OCDS can provide a guide for reviewing and reflecting on current levels of disclosure, and for turning unstructured information into data to analyse. This is the approach pioneered by projects like Budeshi, who started out transcribing documents to data to demonstrate the value that a more data-driven approach could have to procurement monitoring.

  • Bolt-on – OCDS can be used as the target format when exporting data from existing contracting data systems. These might be reporting systems that capture regular monitoring returns on the contracting process, or transactional systems through which procurement is run. Here, the process of mapping existing data to OCDS can often reveal data quality issues in the source systems – and with the right feedback loops, this can lead to not only data publication, but also changes to improve data in future.

  • Built-in – OCDS can be used to inform the design of new systems – providing common shared data models, and a community where extended modelling of data can be discussed. However, it’s important to remember that building new systems is not just about data structures – it’s also about user experience, and right now, the OCDS doesn’t address this.

To my mind, OCDS provide a structured framework that should support use in all these different ways. As we iterate on the standard itself, it’s important we don’t undermine this flexibility – but that instead we use it to establish common ground on which publishers and users can debate issues of data quality. With the standard, those debates should be actionable: but it’s not up the standard itself to settle them.

(4) Contracting is core: but it doesn’t start or end there

Contracting is just one of the government processes that affects how resources are allocated and used. Before contracting starts, budgets are often set, or wide-reaching procurement plans established. During contract implementation, payment processes kick-in. And for the private companies involved in public contracts, there are all sorts of interlocking processes of registration, financing and taxation.

From an architectural perspective it’s important for us to understand the boundaries of the open contracting process, and how it can link up with other processes. For example, whilst OCDS can capture budget information as part of a contracting process (e.g. the amount of budget allocated to that process), it starts stretching the data model to represent a budget process nested within a contracting process.

As one of the break-out groups looking at budget, contract and spend integration noted, the key to joining up data is not putting everything in the same dataset or system, but comes from establishing business processes that ensure common identifiers are used to join up the systems that manage parallel processes.

There’s a lot of work to do before we have easy interoperability between different parts of an overall [accountability architecture](ACSP LINK LINK LINK) – but the biggest issues are of data production and use, rather than of standards and schemas.

(5) It’s time to tidy our terminology

The open contracting community is broad, and, as I recently wrote over here, “the word ‘standard’ means different things to different people.”. So does contracting. And tender. And validation. And assessment. And so-on.

Following recent workshops in London and Argentina, the OCDS team have been thinking a lot about how we tighten up our use of key terms and concepts, establishing a set of draft translation principles and policies, and we’ve been reflecting more on how to also be clearer on ideas like data validity, quality and feedback.

But we also have to recognise that debates over language are laden with power dynamics: and specialist language can be used to impose or exclude. Open contracting should not be about dumbing down complex processes of contracting, but nor should it be able requiring every citizen to learn procurement-speak. Again, for OCDS and other tools designed to support open contracting, we have balancing act: creating boundary objects that help different communities meet in the middle.

The first step towards this is just working out how we’re using words at the moment: checking on current practice, before working out how we can improve.

Gratitude

Asides from sparking a wealth of learning, the other thing an event like #OCGlobal17 does is remind me just how fortunate I am to get to work with such a inspiring network of people: exploring challenging issues with a great collaborative spirit. Thanks all!

The reflections above are more or less fragmentary, and I’m looking forward to working with many of the folk in the picture below to see where the journey takes us next.

Exploring participatory public data infrastructure in Plymouth

[Summary: Slides, notes and references from a conference talk in Plymouth]

Update – April 2020: A book chapter based on this blog post is now published as “Shaping participatory public data infrastructure in the smart city: open data standards and the turn to transparency” in The Routledge Companion to Smart Cities.

Original blog post version below: 

A few months back I was invited to give a presentation to a joint plenary of the ‘Whose Right to the Smart City‘ and ‘DataAche 2017‘ conferences in Plymouth. Building on some recent conversations with Jonathan Gray, I took the opportunity to try and explore some ideas around the concept of ‘participatory data infrastructure’, linking those loosely with the smart cities theme.

As I fear I might not get time to turn it into a reasonable paper anytime soon, below is a rough transcript of what I planned to say when I presented earlier today. The slides are also below.

For those at the talk, the promised references are found at the end of this post.

Thanks to Satyarupa Shekar for the original invite, Katharine Willis and the Whose Right to the Smart Cities network for stimulating discussions today, and to the many folk whose ideas I’ve tried to draw on below.

Participatory public data infrastructure: open data standards and the turn to transparency

In this talk, my goal is to explore one potential strategy for re-asserting the role of citizens within the smart-city. This strategy harnesses the political narrative of transparency and explores how it can be used to open up a two-way communication channel between citizens, states and private providers.

This not only offers the opportunity to make processes of governance more visible and open to scrutiny, but it also creates a space for debate over the collection, management and use of data within governance, giving citizens an opportunity to shape the data infrastructures that do so much to shape the operation of smart cities, and of modern data-driven policy and it’s implementation.

In particular, I will focus on data standards, or more precisely, open data standards, as a tool that can be deployed by citizens (and, we must acknowledge, by other actors, each with their own, sometimes quite contrary interests), to help shape data infrastructures.

Let me set out the structure of what follows. It will be an exploration in five parts, the first three unpacking the title, and then the fourth looking at a number of case studies, before a final section summing up.

  1. Participatory public data infrastructure
  2. Transparency
  3. Standards
  4. Examples: Money, earth & air
  5. Recap

Part 1: Participatory public data infrastructure

Data infrastructure

infrastructure. /?nfr?str?kt??/ noun. “the basic physical and organizational structures and facilities (e.g. buildings, roads, power supplies) needed for the operation of a society or enterprise.” 1

The word infrastructure comes from the latin ‘infra-‘ for below, and structure, meaning structure. It provides the shared set of physical and organizational arrangements upon which everyday life is built.

The notion of infrastructure is central to conventional imaginations of the smart city. Fibre-optic cables, wireless access points, cameras, control systems, and sensors embedded in just about anything, constitute the digital infrastructure that feed into new, more automated, organizational processes. These in turn direct the operation of existing physical infrastructures for transportation, the distribution of water and power, and the provision of city services.

However, between the physical and the organizational lies another form of infrastructure: data and information infrastructure.

(As a sidebar: Although data and information should be treated as analytically distinct concepts, as the boundary between the two concepts is often blurred in the literature, including in discussions of ‘information infrastructures’, and as information is at times used as a super-category including data, I won’t be too strict in my use of the terms in the following).

(That said,) It is by being rendered as structured data that the information from the myriad sensors of the smart city, or the submissions by hundreds of citizens through reporting portals, are turned into management information, and fed into human or machine based decision-making, and back into the actions of actuators within the city.

Seen as a set of physical or digital artifacts, the data infrastructure involves ETL (Extract, Transform, Load) processes, APIs (Application Programming Interfaces), databases and data warehouses, stored queries and dashboards, schema, codelists and standards. Seen as part of a wider ‘data assemblage’ (Kitchin 5) this data infrastructure also involves various processes of data entry and management, of design, analysis and use, as well relationships to other external datasets, systems and standards.

However, if is often very hard to ‘see’ data infrastructure. By their very natures, infrastructures moves into the background, often only ‘visible upon breakdown’ to use Star and Ruhleder’s phrase 2. (For example, you may only really pay attention to the shape and structure of the road network when your planned route is blocked…). It takes a process of “infrastructural inversion” to bring information infrastructures into view 3, deliberately foregrounding the background. I will argue shortly that ‘transparency’ as a policy performs much the same function as ‘breakdown’ in making the contours infrastructure more visible: taking something created with one set of use-cases in mind, and placing it in front of a range of alternative use-cases, such that its affordances and limitations can be more fully scrutinized, and building on that scrutiny, it’s future development shaped. But before we come to that, we need to understand the extent of ‘public data infrastructure’ and the different ways in which we might understand a ‘participatory public data infrastructure’.

Public data infrastructure

There can be public data without a coherent public data infrastructure. In ‘The Responsive City’ Goldsmith and Crawford describe the status quo for many as “The century-old framework of local government – centralized, compartmentalized bureaucracies that jealously guard information…” 4. Datasets may exist, but are disconnected. Extracts of data may even have come to be published online in data portals in response to transparency edicts – but it exists as islands of data, published in different formats and structures, without any attention to interoperability.

Against this background, initiatives to construct public data infrastructure have sought to introduce shared technology, standards and practices that provide access to a more coherent collection of data generated by, and focusing on, the public tasks of government.

For example, in 2012, Denmark launched their ‘Basic Data’ programme, looking to consolidate the management of geographic, address, property and business data across government, and to provide common approaches to data management, update and distribution 6. In the European Union, the INSPIRE Directive and programme has been driving creation of a shared ‘Spatial Data Infrastructure’ since 2007, providing reference frameworks, interoperability rules, and data sharing processes. And more recently, the UK Government has launched a ‘Registers programme’ 8 to create centralized reference lists and identifiers of everything from countries to government departments, framed as part of building governments digital infrastructure. In cities, similar processes of infrastructure building, around shared services, systems and standards are taking place.

The creation of these data infrastructures can clearly have significant benefits for both citizens and government. For example, instead of citizens having to share the same information with multiple services, often in subtly different ways, through a functioning data infrastructure governments can pick up and share information between services, and can provide a more joined up experience of interacting with the state. By sharing common codelists, registers and datasets, agencies can end duplication of effort, and increase their intelligence, drawing more effectively on the data that the state has collected.

However, at the same time, these data infrastructures tend to have a particularly centralizing effect. Whereas a single agency maintaining their own dataset has the freedom to add in data fields, or to restructure their working processes, in order to meet a particular local need – when that data is managed as part of a centralized infrastructure, their ability to influence change in the way data is managed will be constrained both by the technical design and the institutional and funding arrangements of the data infrastructure. A more responsive government is not only about better intelligence at the center, it is also about autonomy at the edges, and this is something that data infrastructures need to be explicitly designed to enable, and something that they are generally not oriented towards.

In “Roads to Power: Britain Invents the Infrastructure State” 10, Jo Guldi uses a powerful case study of the development of the national highways networks to illustrate the way in which the design of infrastructures shapes society, and to explore the forces at play in shaping public infrastructure. When metaled roads first spread out across the country in the eighteenth century, there were debates over whether to use local materials, easy to maintain with local knowledge, or to apply a centralized ‘tarmacadam’ standard to all roads. There were questions of how the network should balance the needs of the majority, with road access for those on the fringes of the Kingdom, and how the infrastructure should be funded. This public infrastructure was highly contested, and the choices made over it’s design had profound social consequences. Jo uses this as an analogy for debates over modern Internet infrastructures, but it can be equally applied to explore questions around an equally intangible public data infrastructure.

If you build roads to connect the largest cities, but leave out a smaller town, the relative access of people in that town to services, trade and wider society is diminished. In the same way, if your data infrastructure lack the categories to describe the needs of a particular population, their needs are less likely to be met. Yet, that town connected might also not want to be connected directly to the road network, and to see it’s uniqueness and character eroded; much like some groups may also want to resist their categorization and integration in the data infrastructure in ways that restrict their ability to self-define and develop autonomous solutions, in the face of centralized data systems that are necessarily reductive.

Alongside this tension between centralization and decentralization in data infrastructures, I also want to draw attention to another important aspect of public data infrastructures. That is the issue of ownership and access. Increasingly public data infrastructures may rely upon stocks and flows of data that are not publicly owned. In the United Kingdom, for example, the Postal Address File, which is the basis of any addressing service, was one of the assets transferred to the private sector when Royal Mail was sold off. The Ordnance Survey retains ownership and management of the Unique Property Reference Number (UPRN), a central part of the data infrastructure for local public service delivery, yet access to this is heavily restricted, and complex agreements govern the ability of even the public sector to use it. Historically, authorities have faced major challenges in relation to ‘derived data’ from Ordnance Survey datasets, where the use of proprietary mapping products as a base layer when generating local records ‘infects’ those local datasets with intellectual property rights of the proprietary dataset, and restricts who they can be shared with. Whilst open data advocacy has secured substantially increased access to many publicly owned datasets in recent years, when the datasets the state is using are privately owned in the first place, and only licensed to the state, the potential scope for public re-use and scrutiny of the data, and scrutiny of the policy made on the basis of it, is substantially limited.

In the case of smart cities, I suspect this concern is likely to be particularly significant. Take transit data for example: in 2015 Boston, Massachusetts did a deal with Uber to allow access to data from the data-rich transportation firm to support urban planning and to identify approaches to regulation. Whilst the data shared reveals something of travel times, the limited granularity rendered it practically useless for planning purposes, and Boston turned to senate regulations to try and secure improved data 9. Yet, even if the city does get improved access to data about movements via Uber and Lyft in the city – the ability of citizens to get involved in the conversations about policy from that data may be substantially limited by continued access restrictions on the data.

With the Smart City model often involving the introduction of privately owned sensors networks and processes, the extent to which the ‘data infrastructure for public tasks ceases to have the properties that we will shortly see are essential to a ‘participatory public data infrastructure’ is a question worth paying attention to.

Participatory public data infrastructure

I will posit then that the grown of public data infrastructures is almost inevitable. But the shape they take is not. I want, in particular then, to examine what it would mean to have a participatory public data infrastructure.

I owe the concept of a ‘participatory public data infrastructure’ in particular to Jonathan Gray ([11], [12], [13]), who has, across a number of collaborative projects, sought to unpack questions of how data is collected and structured, as well as released as open data. In thinking about the participation of citizens in public data, we might look at three aspects:

  1. Participation in data use
  2. Participation in data production
  3. Participation in data design

And, seeing these as different in kind, rather than different in degree, we might for each one deploy Arnstein’s ladder of participation [14] as an analytical tool, to understand that the extent of participation can range from tokenism through to full shared decision making. As for all participation projects, we must also ask the vitally important question of ‘who is participating?’.

At the bottom-level ‘non-participation’ runs of Arnstein’s ladder we could see a data infrastructure that captures data ‘about’ citizens, without their active consent or involvement, that excludes them from access to the data itself, and then uses the data to set rules, ‘deliver’ services, and enact policies over which citizens have no influence in either their design of delivery. The citizen is treated as an object, not an agent, within the data infrastructure. For some citizens contemporary experience, and in some smart city visions, this description might not be far from a direct fit.

By contrast, when citizens have participation in the use of a data infrastructure they are able to make use of public data to engage in both service delivery and policy influence. This has been where much of the early civic open data movement placed their focus, drawing on ideas of co-production, and government-as-a-platform, to enable partnerships or citizen-controlled initiatives, using data to develop innovative solutions to local issues. In a more political sense, participation in data use can remove information inequality between policy makers and the subjects of that policy, equalizing at least some of the power dynamic when it comes to debating policy. If the ‘facts’ of population distribution and movement, electricity use, water connections, sanitation services and funding availability are shared, such that policy maker and citizen are working from the same data, then the data infrastructure can act as an enabler of more meaningful participation.

In my experience though, the more common outcome when engaging diverse groups in the use of data, is not an immediate shared analysis – but instead of a lot of discussion of gaps and issues in the data itself. In some cases, the way data is being used might be uncontested, but the input might turn out to be misrepresenting the lived reality of citizens. This takes us to the second area of participation: the ability to not jusT take from a dataset, but also to participate in dataset production. Simply having data collected from citizens does not make a data infrastructure participatory. That sensors tracked my movement around an urban area, does not make me an active participant in collecting data. But by contrast, when citizens come together to collect new datasets, such as the water and air quality datasets generated by sensors from Public Lab 15, and are able to feed this into the shared corpus of data used by the state, there is much more genuine participation taking place. Similarly, the use of voluntary contributed data on Open Street Map, or submissions to issue-tracking platforms like FixMyStreet, constitute a degree of participation in producing a public data infrastructure when the state also participates in use of those platforms.

It is worth noting, however, that most participatory citizen data projects, whether concerned with data use of production, are both patchy in their coverage, and hard to sustain. They tend to offer an add-on to the public data infrastructure, but to leave the core substantially untouched, not least because of the significant biases that can occur due to inequalities of time, hardware and skills to be able to contribute and take part.

If then we want to explore participation that can have a sustainable impact on policy, we need to look at shaping the core public data infrastructure itself – looking at the existing data collection activities that create it, and exploring whether or not the data collected, and how it is encoded, serves the broad public interest, and allows the maximum range of democratic freedom in policy making and implementation. This is where we can look at a participatory data infrastructure as one that enables citizens (and groups working on their behalf) to engage in discussions over data design.

The idea that communities, and citizens, should be involved in the design of infrastructures is not a new one. In fact, the history of public statistics and data owes a lot to voluntary social reform focused on health and social welfare collecting social survey data in the eighteenth and nineteenth centuries to influence policy, and then advocating for government to take up ongoing data collection. The design of the census and other government surveys have long been sources of political contention. Yet, with the vast expansion of connected data infrastructures, which rapidly become embedded, brittle and hard to change, we are facing a particular moment at which increased attention is needed to the participatory shaping of public data infrastructures, and to considering the consequences of seemingly technical choices on our societies in the future.

Ribes and Baker [16], in writing about the participation of social scientists in shaping research data infrastructures draw attention to the aspect of timing: highlighting the limited window during which an infrastructure may be flexible enough to allow substantial insights from social science to be integrated into its development. My central argument is that transparency, and the move towards open data, offers a key window within which to shape data infrastructures.

Part 2: Transparency

transparency /tran?spar(?)nsi/ noun “the quality of being done in an open way without secrets” 21

Advocacy for open data has many distinct roots: not only in transparency. Indeed, I’ve argued elsewhere that it is the confluence of many different agendas around a limited consensus point in the Open Definition that allowed the breakthrough of an open data movement late in the last decade [17] [18]. However, the normative idea of transparency plays an important roles in questions of access to public data. It was a central part of the framing of Obama’s famous ‘Open Government Directive’ in 2009 20, and transparency was core to the rhetoric around the launch of data.gov.uk in the wake of a major political expenses scandal.

Transparency is tightly coupled with the concept of accountability. When we talk about government transparency, it is generally as part of government giving account for it’s actions: whether to individuals, or to the population at large via the third and fourth estates. To give effective account means it can’t just make claims, it has to substantiate them. Transparency is a tool allowing citizens to exercise control over their governments.

Sweden’s Freedom of the Press law from 1766 were the first to establish a legal right to information, but it was a slow burn until the middle of the last century, when ‘right to know’ statutes started to gather pace such that over 100 countries now have Right to Information laws in place. Increasingly, these laws recognize that transparency requires not only access to documents, but also access to datasets.

It is also worth noting that transparency has become an important regulatory tool of government: where government may demand transparency off others. As Fung et. al argue in ‘Full Disclosure’, governments have turned to targeted transparency as a way of requiring that certain information (including from the private sector) is placed in the public domain, with the goal of disciplining markets or influencing the operation of marketized public services, by improving the availability of information upon which citizens will make choices [19].

The most important thing to note here is that demands for transparency are often not just about ‘opening up’ a dataset that already exists – but ultimately are about developing an account of some aspect of public policy. To create this account might require data to be connected up from different silos, and may required the creation of new data infrastructures.

This is where standards enter the story.

Part 3: Standards

standard /?stand?d/ noun

something used as a measure, norm, or model in [comparative] evaluations.

The first thing I want to note about ‘standards’ is that the term is used in very different ways by different communities of practice. For a technical community, the idea of a data standard more-or-less relates to a technical specification or even schema, by which the exact way that certain information should be represented as data is set out in minute detail. To assess if data ‘meets’ the standard is a question of how the data is presented. For a policy audience, talk of data standards may be interpreted much more as a question of collection and disclosure norms. To assess if data meets the standard here is more a question of what data is presented. In practice, these aspects interrelate. With anything more than a few records, to assess ‘what’ has been disclosed requires processing data, and that requires it to be modeled according to some reasonable specification.

The second thing I want to note about standards is that they are highly interconnected. If we agree upon a standard for the disclosure of government budget information, for example, then in order to produce data to meet that standard, government may need to check that a whole range of internal systems are generating data in accordance with the standard. The standard for disclosure that sits on the boundary of a public data infrastructure can have a significant influence on other parts of that infrastructure, or its operation can be frustrated when other parts of the infrastructure can’t produce the data it demands.

The third thing to note is that a standard is only really a standard when it has multiple users. In fact, the greater the community of users, the stronger, in effect, the standard is.

So – with these points in mind, let’s look at how a turn to transparency and open data has created both pressure for application of data standards, and an opening for participatory shaping of data infrastructures.

One of the early rallying cries of the open data movement was ‘Raw Data Now’. Yet, it turns out raw data, as a set of database dumps of selected tables from the silo datasets of the state does not always produce effective transparency. What it does do, however, is create the start of a conversation between citizen, private sector and state over the nature of the data collected, held and shared.

Take for example this export from a council’s financial system in response to a central government policy calling for transparency on spend over £500.

Service Area BVA Cop ServDiv Code Type Code Date Transaction No. Amount Revenue / Capital Supplier
Balance Sheet 900551 Insurance Claims Payment (Ext) 47731 31.12.2010 1900629404 50,000.00 Revenue Zurich Insurance Co
Balance Sheet 900551 Insurance Claims Payment (Ext) 47731 01.12.2010 1900629402 50,000.00 Revenue Zurich Insurance Co
Balance Sheet 933032 Other income 82700 01.12.2010 1900632614 -3,072.58 Revenue Unison Collection Account
Balance Sheet 934002 Transfer Values paid to other schemes 11650 02.12.2010 1900633491 4,053.21 Revenue NHS Pensions Scheme Account
Balance Sheet 900601 Insurance Claims Payment (Ext) 47731 06.12.2010 1900634912 1,130.54 Revenue Shires (Gloucester) Ltd
Balance Sheet 900652 Insurance Claims Payment (Int) 47732 06.12.2010 1900634911 1,709.09 Revenue Bluecoat C Of E Primary School
Balance Sheet 900652 Insurance Claims Payment (Int) 47732 10.12.2010 1900637635 1,122.00 Revenue Christ College Cheltenham

It comes from data generated for one purpose (the council’s internal financial management), now being made available for another purpose (external accountability), but that might also be useful for a range of further purposes (companies looking to understand business opportunities; other council’s looking to benchmark their spending, and so-on). Stripped of its context as part of internal financial systems, the column headings make less sense: what is BVA COP? Is the date the date of invoice? Or of payment? What does each ServDiv Code relate to? The first role of any standardization is often to document what the data means: and in doing so, to surface unstated assumptions.

But standardization also plays a role in allowing the emerging use cases for a dataset to be realized. For example, when data columns are aligned comparison across council spending is facilitated. Private firms interested in providing such comparison services may also have a strong interest in seeing each of the authorities providing data doing so to a common standard, to lower their costs of integrating data from each new source.

If standards are just developed as the means of exchanging data between government and private sector re-users of the data, the opportunities for constructing a participatory data infrastructure are slim. But when standards are explored as part of the transparency agenda, and as part of defining both the what and the how of public disclosure, such opportunities are much richer.

When budget and spend open data became available in Sao Paulo in Brazil, a research group at University of Sao Paulo, led by Gisele Craviero, explored how to make this data more accessible to citizens at a local level. They found that by geocoding expenditure, and color coding based on planned, committed and settled funds, they could turn the data from impenetrable tables into information that citizens could engage with. More importantly, they argue that in engaging with government around the value of geocoded data “moving towards open data can lead to changes in these underlying and hidden process [of government data creation], leading to shifts in the way government handles its own data” [22]

The important act here was to recognize open data-enabled transparency not just as a one-way communication from government to citizens, but as an invitation for dialog about the operation of the public data infrastructure, and an opportunity to get involved – explaining that, if government took more care to geocode transactions in its own systems, it would not have to wait for citizens to participate in data use and to expend the substantial labour on manually geocoding some small amount of spending, but instead the opportunity for better geographic analysis of spending would become available much more readily inside and outside the state.

I want to give three brief examples of where the development, or not, of standards is playing a role in creating more participatory data infrastructures, and in the process to draw out a couple of other important aspects of thinking about transparency and standardization as part of the strategic toolkit for asserting citizen rights in the context of smart cities.

Part 4: Examples

Contracts

My first example looks at contracts for two reasons. Firstly, it’s an area I’ve been working on in depth over the last few years, as part of the team creating and maintaining the Open Contracting Data Standard. But, more importantly, its an under-explored aspect of the smart city itself. For most cities, how transparent is the web of contracts that establishes the interaction between public and private players? Can you easily find the tenders and awards for each component of the new city infrastructure? Can you see the terms of the contracts and easily read-up on who owns and controls each aspect of emerging public data infrastructure? All too often the answer to these questions is no. Yet, when it comes to procurement, the idea of transparency in contracting is generally well established, and global guidance on Public Private Partnerships highlights transparency of both process and contract documents as an essential component of good governance.

The Open Contracting Data Standard emerged in 2014 as a technical specification to give form to a set of principles on contracting disclosure. It was developed through a year-long process of research, going back and forth between a focus on ‘data supply’ and understanding the data that government systems are able to produce on their contracting, and ‘data demand’, identifying a wide range of user groups for this data, and seeking to align the content and structure of the standard with their needs. This resulted in a standard that provides a framework for publication of detailed information at each stage of a contracting process, from planning, through tender, award and signed contract, right through to final spending and delivery.

Meeting this standard in full is quite demanding for authorities. Many lack existing data infrastructures that provide common identifiers across the whole contracting process, and so adopting OCDS for data disclosure may involve some elements of update to internal systems and processes. The transparency standard has an inwards effect, shaping not only the data published, but the data managed. In supporting implementation of OCDS, we’ve also found that the process of working through the structured publication of data often reveals as yet unrecognized data quality issues in internal systems, and issues of compliance with existing procurement policies.

Now, two of the critiques that might be offered of standards is that, as highly technical objects their development is only open to participation from a limited set of people, and that in setting out a uniform approach to data publication, they are a further tool of centralization. Both these are serious issues.

In the Open Contracting Data Standard we’ve sought to navigate them by working hard on having an open governance process for the standard itself, and using a range of strategies to engagement people in shaping the standard, including workshops, webinars, peer-review processes and presenting the standard in a range of more accessible formats. We’re also developing an implementation and extensions model that encourages local debate over exactly which elements of the overall framework should be prioritized for publication, whilst highlighting the fields of data that are needed in order to realize particular use-cases.

This highlights an important point: standards like OCDS are more than the technical spec. There is a whole process of support, community building, data quality assurance and feedback going on to encourage data interoperability, and to support localization of the standard to meet particular needs.

When standards create the space, then other aspects of a participatory data infrastructure are also enabled and facilitated. A reliable flow of data on pipeline contracts may allow citizens to scrutinize the potential terms of tenders for smart city infrastructure before contracts are awarded and signed, and an infrastructure with the right feedback mechanisms could ensure, for example, that performance-based payments to providers are properly influenced by independent citizen input.

The thesis here is one of breadth and depth. A participatory developed open standard allows a relatively small-investment intervention to shape a broad section of public data infrastructure, influencing the internal practice of government and establishing the conditions for more ad-hoc deep-dive interventions, that allow citizens to use that data to pursue particular projects of change.

Earth

The second example explores this in the context of land. Who owns the smart city?

The Open Data Index and Open Data Barometer studies of global open data availability have had a ‘Land Ownership’ category for a number of years, and there is a general principle that land ownership information should, to some extent, be public. However, exactly what should be published is a tricky question. An over-simplified schema might ignore the complex realities of land rights, trying to reduce a set of overlapping claims to a plot number and owner. By contrast, the narrative accounts of ownership that often exist in the documentary record may be to complex to render as data [24]. In working on a refined Open Data Index category, the Cadasta Foundation 23 noted that opening up property owners names in the context of a stable country with functioning rule of law “has very different risks and implications than in a country with less formal documentation, or where dispossession, kidnapping, and or death are real and pervasive issues” 23.

The point here is that a participatory process around the standards for transparency may not, from the citizen perspective, always drive at more disclosure, but that at times, standards may also need to protect the ‘strategic invisibility’ of marginalized groups [25]. In the United Kingdom, although individual titles can be bought for £3 from the Land Registry, no public dataset of title-holders is available. However, there are moves in place to establish a public dataset of land owned by private firms, or foreign owners, coming in part out of an anti-corruption agenda. This fits with the idea that, as Sunil Abraham puts it, “privacy should be inversely proportional to power” 26.

Central property registers are not the only source of data relevant to the smart city. Public authorities often have their own data on public assets. A public conversation on the standards needed to describe this land, and share information about it, is arguable overdue. Again looking at the UK experience, the government recently consulted on requiring authorities to record all information on their land assets through the Property Information Management system (ePIMS): centralizing information on public property assets, but doing so against a reductive schema that serves central government interests. In the consultation on this I argued that, by contrast, we need an approach based on a common standard for describing public land, but that allows local areas the freedom to augment a core schema with other information relevant to local policy debates.

Air

From the earth, let us turn very briefly to the air. Air pollution is a massive issue, causing millions on premature deaths worldwide every year. It is an issue that is particularly acute in urban areas. Yet, as the Open Data Institute note “we are still struggling to ‘see’ air pollution in our everyday lives” 27. They report the case of decision making on a new runway at Heathrow Airport, where policy makers were presented with data from just 14 NO2 sensors. By contrast, a network of citizen sensors provided much more granular information, and information from citizen’s gardens and households, offering a different account from those official sensors by roads or in fields.

Mapping the data from official government air quality sensors reveals just how limited their coverage is: and backs up the ODI’s calls for a collaborative, or participatory, data infrastructure. In a 2016 blog post, Jamie Fawcett describes how:

“Our current data infrastructure for air quality is fragmented. Projects each have their own goals and ambitions. Their sensor networks and data feeds often sit in silos, separated by technical choices, organizational ambition and disputes over data quality and sensor placement. The concerns might be valid, but they stand in the way of their common purpose, their common goals.”

He concludes “We need to commit to providing real-time open data using open standards.”

This is a call for transparency by both public and private actors: agreeing to allow re-use of their data, and rendering it comparable through common standards. The design of such standards will need to carefully balance public and private interests, and to work out how the costs of making data comparable will fall between data publishers and users.

Part 5: Recap

So, to briefly recap:

  • I want to draw attention to the data infrastructures of the smart city and the modern state;
  • I’ve suggested that open data and transparency can be powerful tools in performing the kind of infrastructural inversion that brings the context and history of datasets into view and opens them up to scrutiny;
  • I’ve furthermore argued that transparency policy opens up an opportunity for a two-way dialogue about public data infrastructures, and for citizen participation not only in the use and production of data, but also in setting standards for data disclosure;
  • I’ve then highlighted how standards for disclosure don’t just shape the data that enters the public domain, but they also have an upwards impact on the shape of the public data infrastructure itself.

Taken together, this is a call for more focus on the structure and standardization of data, and more work on exploring the current potential of standardization as a site of participation, and an enabler of citizen participation in future.

If you are looking for a more practical set of takeaways that flow from all this, let me offer a set of questions that can be asked of any smart cities project, or indeed, any data-rich process of governance:

  • (1) What information is pro-actively published, or can be demanded, as a result of transparency and right to information policies?
  • (2) What does the structure of the data reveal about the process/project it relates to?
  • (3) What standards might be used to publish this data?
  • (4) Do these standards provide the data I, or other citizens, need to be empowered in relevant to this process/project?
  • (5) Are these open standards? Whose needs were they designed to serve?
  • (6) Can I influence these standards? Can I afford not to?

References

1: https://www.google.co.uk/search?q=define%3Ainfrastructure, accessed 17th August 2017

2: Star, S., & Ruhleder, K. (1996). Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces. Information Systems Research, 7(1), 111–134.

3: Bowker, G. C., & Star, S. L. (2000). Sorting Things Out: Classification and Its Consequences. The MIT Press.

4: Goldsmith, S., & Crawford, S. (2014). The responsive city. Jossey-Bass.

5: Kitchin, R. (2014). The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences. SAGE Publications.

6: The Danish Government. (2012). Good Basic Data for Everyone – a Driver for Growth and Efficiency, (October 2012)

7: Bartha, G., & Kocsis, S. (2011). Standardization of Geographic Data: The European INSPIRE Directive. European Journal of Geography, 22, 79–89.

10: Guldi, J. (2012). Roads to power: Britain invents the infrastructure state.

[11]: Gray, J., & Davies, T. (2015). Fighting Phantom Firms in the UK : From Opening Up Datasets to Reshaping Data Infrastructures?

[12]: Gray, J., & Tommaso Venturini. (2015). Rethinking the Politics of Public Information: From Opening Up Datasets to Recomposing Data Infrastructures?

[13]: Gray, J. (2015). DEMOCRATISING THE DATA REVOLUTION: A Discussion Paper

[14]: Arnstein, S. R. (1969). A ladder of citizen participation. Journalof the American Institute of Planners, 34(5), 216–224.

[16]: Ribes, D., & Baker, K. (2007). Modes of social science engagement in community infrastructure design. Proceedings of the 3rd Communities and Technologies Conference, C and T 2007, 107–130.

[17]: Davies, T. (2010, September 29). Open data, democracy and public sector reform: A look at open government data use from data.gov.uk.

[18]: Davies, T. (2014). Open Data Policies and Practice: An International Comparison.

[19]: Fung, A., Graham, M., & Weil, D. (2007). Full Disclosure: The Perils and Promise of Transparency (1st ed.). Cambridge University Press.

[22]: Craveiro, G. S., Machado, J. A. S., Martano, A. M. R., & Souza, T. J. (2014). Exploring the Impact of Web Publishing Budgetary Information at the Sub-National Level in Brazil.

[24]: Hetherington, K. (2011). Guerrilla auditors: the politics of transparency in neoliberal Paraguay. London: Duke University Press.

[25]: Scott, J. C. (1987). Weapons of the Weak: Everyday Forms of Peasant Resistance.

Open data for tax justice: the real design challenge is social

[Summary: Thinking aloud about a pragmatic / humanist approach to data infrastructure building]

Stephen Abbott Pugh of Open Knowledge International has just blogged about the Open Data for Tax Justice ‘design sprint’ that took place in London on Monday and Tuesday. I took part in the first day and a half of the workshop, and found myself fairly at-odds with the approach being taken that focussed narrowly on the data-pipelines based creation of a centralised dataset, and that appeared to create barriers rather than bridges between data and domain experts. Rather than the rethink the approach, as I would argue is needed, the Open Knowledge write up appears to show the Open Data for Tax Justice project heading further down this flawed path.

In this post, I’m offering an (I hope) constructive critique of the approach, trying to draw out some more general principles that might inform projects to create more participatory data infrastructures.

The context

As the OKI post relates:

“Country-by-country reporting (CBCR) is a transparency mechanism which requires multinational corporations to publish information about their economic activities in all of the countries where they operate. This includes information on the taxes they pay, the number of people they employ and the profits they report.”

Country by Country reporting has been a major ask of tax justice campaigners since the early 2000s, in order to address tax avoidance by multi-national companies who shift their profits around the world through complex corporate structures and internal transfers. CBCR got a major boost in 2013 with the launch of reporting requirements for EU Banks to publicly disclose Country by Country reports under the CRD IV regulations. In the extractives sector, campaigners have also secured regulations requiring disclosure of tax and licensing payments to government on a project-by-project basis.

Although in the case of UK extractives firms, reporting is taking place to companies house as structured data, with an API available to access reports, for EU Banks, reporting is predominantly in the form of tables at the back of PDF format company reports.

If campaigners are successful, public reporting will be extended to all EU multinationals, holding out the prospect of up to 6000 more annual reports that can provide a breakdown of turnover, profit, tax and employees country-by-country. If the templates for disclosure are based on existing OECD models for private exchange between tax authorities, the data may also include information on the different legal entities that make a corporate group, important for public understanding the structure of the corporate world.

Earlier this year, a report from Alex Cobham, Jonathan Gray and Richard Murphey set out a number of use-cases for such data, making the case that “a global public database on the tax contributions and economic activities of multinational companies” would be an asset for a wide range of users, from journalists, civil society and investors.

Sprinting with a data-pipelines hammer

This week’s design sprint focussed particularly on ‘data extraction’, developing a set of data pipeline scripts and processes that involve downloading a report PDF, marking up the tables where Country by Country data is stored, describing what each column contains using YAML, and then committing this to GitHub where the process can then be replicably run using datapipeline commands. Then, with the data extracted, it can be loaded into an SQL database, and explored by writing queries or building simple charts. It’s a technically advanced approach, and great for ensuring replicability of data extraction.

But, its also an approach that ultimately entirely misses the point, ignoring the social process of data production, creating technical barriers instead of empowering contributors and users, and offering nothing for campaigners who want to ensure that better data is produced ‘at source’ by companies.

Whilst the OKI blog post reports that “The Open Data for Tax Justice network team are now exploring opportunities for collaborations to collect and process all available CRD IV data via the pipeline and tools developed during our sprint.” I want to argue for a refocussed approach, based around a much closer look at the social dynamics of data creation and use.

An alternative approach: crafting collaborations

I’ve tried below to unpack a number of principles that might guide that alternative approach:

Principle 1: Letting people use their own tools

Any approach that involves downloading, installing, signing-up to, configuring or learning new software in order to create or use data is likely to exclude a large community of potential users. If the data you are dealing with is tabular: focus on spreadsheets.

More technical users can transform data into database formats when the questions they want to answer require the additional power that brings, but it is better if the starting workflow is configured to be accessible to the largest number of likely users.

Back in October I put together a rough prototype of a Google spreadsheets based transcription tool for Country by Country reports, that needed just copy-and-paste of data, and a few selections from validated drop-down lists to go from PDFs to normalised data – allowing a large user community to engage directly with the data, with almost zero learning curve.

The only tool this approach needs to introduce is something like tabula or PDFTables to convert from PDF to Excel or CSV: but in this workflow the data comes right back to the user to be able to work with it after it has been converted, rather than being taken away from them into a longer processing pipeline. Plus, it brings the benefit of raising awareness of data extraction from PDF that the user can adopt for other projects in future, and allowing the user to work-around failed conversions using a manual transcription approach if they need to.

(Sidenote: from discussions, I understand that one of the reasons the OKI team made their technical choice was from envisaging the primary users as ‘non-experts’ who would engage in crowdsourcing transcriptions of PDF reports. I think this is both highly optimistic, and relies on a flawed analysis of the relatively small scale of the crowdsourcing task in terms of a few 1000 reports a year, and the potential benefits of involving a more engaged group of contributors in creating a civil society database)

Principle 2: Aim for instant empowerment

One of the striking things about Country by Country reporting data is how simple it ultimately is. The CRD IV disclosures contain just a handful of measures (turnover, pre-tax profits, tax paid, number of employees), a few dimensions (company name, country, year), and a range of annotations in footnotes or explanations. The analysis that can be done with this is data is similarly simple – yet also very powerful. Being able to go from a PDF table of data, to a quick view of the ratios between turnover and tax, or profit and employees for a country can quickly highlight areas to investigate for profit-shifting and tax-avoidance behaviour.

Calculating these ratios is possible almost as soon as you have data in a spreadsheet form. In fact, a well set up template could calculate them directly, or the user with basic ability to write formula could fill in the columns they need.

Many of the use-cases for Country by Country reports are based not on aggregation across hundreds of firms, but on simply understanding the behaviour of one or two firms. Investigators and researchers often have firms they are particularly interested in, and where the combination of simple data, and their contextual knowledge, can go a long way.

Principle 3: Don’t drop context

On the topic of context: all those footnotes and explanations in company reports are an important part of the data. They might not be computable, or easy to query against, but in the data explorations that took place on Monday and Tuesday I was struck by how much the tax justice experts were relying not only on the numerical figures to find stories, but also on the explanations and other annotations from reports.

The data pipelines approach dropped these annotations (and indeed dropped anything that didn’t fit into it’s schema). An alternative approach would work from the principle that, as far as possible, nothing of the source should be thrown away – and structure should be layered on top of the messy reality of accounting judgements and decisions.

Principle 4: Data making is meaning-making

A lot of the analysis of Country by Country reporting data is about look for outliers. But data outliers and data errors can look pretty similar. Instead of trying to separate the process of data preparation and analysis, these two need to be brought closer together.

Creating a shared database of tax disclosures will involve not only processes of data extraction, but also processes of validation and quality control. It will require incentives for contributors, and will require attention to building a community of users.

Some of the current structured data available from Country by Country reports has been transcribed by University students as part of their classes – where data was created as a starting point for a close feedback loop of data analysis. The idea of ‘frictionless data’ makes sense when it comes to getting a list of currency codes, but when it comes to understanding accounts, some ‘friction’ of social process can go a long way to getting reliable data, and building a community of practice who understand the data in more depth.

Principle 5: Standards support distributed collaboration

One of the difficulties in using the data mentioned above, prepared by a group of students, was that it had been transcribed and structured to solve the particular analytical problem of the class, and not against any shared standard for identifying countries, companies or the measures being transcribed.

The absence of agreement on key issues such as codelists for tax jurisdictions, company identifiers, codes and definitions of measures, and how to handle annotations and missing data means that the data that is generated by different researchers, or even different regulatory regimes, is not comparable, and can’t be easily combined.

The data pipelines approach is based on rendering data comparable through a centralised infrastructure. In my experience, such approaches are brittle, particularly in the context of voluntary collaboration, and they tend to create bottlenecks for data sharing and innovation. By contrast, an approach based on building light-weight standards can support a much more distributed collaboration approach – in which different groups can focus first on the data that is of most interest to them (for example, national journalists focussing on the tax record of the top-10 companies in their jurisdiction), easily contributing data to a common pool later when their incentives are aligned.

Campaigners also need to be armed with use-case backed proposals for how disclosures should be structured in order to push for the best quality disclosure regimes

What’s the difference?

Depending on your viewpoint, the approach I’ve started to set out above might look more technically ‘messy’ – but I would argue it is more in-tune with the social realities of building a collaborative dataset of company tax disclosures.

Fundamentally (with the exception perhaps of standard maintenance, although that should be managed as a multi-stakeholder project long-term) – it is much more decentralised. This is in line with the approach in the Open Contracting Data Standard, where the Open Contracting Partnership have stuck well to their field-building aspirations, and where many of the most interesting data projects emerge organically at the edge of the network, only later feeding into cross-collaboration.

Even then, this sketch of an alternative technical approach above is only part of the story in building a better data-foundation for action to address corporate tax avoidance. There will still be a lot of labour to create incentives, encourage co-operation, manage data quality, and build capacity to work with data. But better we engage with that labour, than spending our efforts chasing after frictionless dreams of easily created perfect datasets.

The ongoing secrecy saga of Javelin Park: Ernst and Young Value for Money Analysis

[Summary: the latest in Gloucestershire County Council’s Javelin Park secrecy saga (read up on recent episodes here, here and here)]

In the Information Tribunal ruling EA/2015/0254-6 (which led to the provision of a mostly unredacted copy of the 2013 UBB Javelin Park Incinerator Contract), paragraph 27 contains a reference to reports produced for Cabinet by Ernst and Young that provide the basis for the high estimated cancellation cost of the Javelin Park Incinerator.

I requested a copy of these documents from Gloucestershire County Council (GCC) in an FOI request, and following a long review process, have been provided with a heavily redacted copy of the Ernst and Young report has been provided under the Environment Information Regulations (EIRs).

What can we learn from the redacted copy?

The report provides updated Value for Money and Affordability analysis for the Javelin Park Incinerator contract. It was provided to the Council on 5th November 2015, ahead of the Cabinet approving a second ‘Financial Close’ of the Javelin Park Public Private Partnership project at their meeting of 11th November 2015.

This updates many of the figures given in the 2012 Annex 4 ‘Resource Implications’ that was provided in a fully unredacted form following the Information Tribunal ruling. It also provides a number of insights into the actions of the council to inject additional funding into the project.

Whilst the redactions mean there is litle new financial information here on which to update an understanding of the project Value for Money, I did take note of the following:

  • Due to the planning delays, a ‘Revised Project Planning’ (RPP) process was triggered. This allows for various costs and figures in the contract to be updated (See the 2013 contract §3.3). §5.5 of the report indicates that there are updated tonnage prices in force under the Revised Project Plan, with the prose suggesting these have increased. The prose also suggests that anticipated third-party waste revenues have decreased.

  • The report calculates the cost of a Force Majeure Planning Failure Termination. In November 2015 planning approval was fully in place, so this would have been on the basis of GCC excercising their right to turn down the Revised Project Plan (RPP) from UBB.

  • The report does not calculate the cost of a ‘Voluntary Authority Termination’ (the council choosing not to proceed with construction), but instead states that it “would anticipate a sum in excess of £100m”.

  • §4 of the Ernst and Young report states that: “any decision to terminate and pursue a landfill alternative would require a termination payment to UBB to meet costs already incurred. This cost, amounting to c£60m (as set out in Appendix A) has been added to the cost of the Landfill Alternative.” Appendix A is heavily redacted, so it is not possible to identify the basis for this figure, or why this figure of £60m is lower than the sum ‘anticipated in excess of £100m’. However, this could be the source of the £60m – £100m cancellation cost estimates cited by Cabinet members.

  • In the ‘Force Majeure Cancellation Costs’ calculations in Appendix 1, under sub-contractor breakage and redundancy costs, Ernst and Young note that no evidence is held on the actual costs expended by UBB to date, nor the sub-contract breakage costs that would actually be incurred.

  • As of November 2015, Eversheds had produced legal advice to the council on Procurement risks including risk of challenge (p. 3)

  • The affordability analysis (§1.2) “identifies that without the capital contribution [£17m] the Project is in breach of the Council’s affordability limit until 2024 but there after falls inside the affordability limit” and introducing the £17m capital contribution moves affordability to 2022.

At the November 2015 Cabinet meeting the Cabinet claimed savings from the project of £153m, based on the difference between a Landfill base scenario of £522m and project cost of £399m (once a £13m financial contribution from the council had been made). This assumes a waste flow of 60% recycling. The savings substantially erode (a c. 60% decrease in Net Present Value) with lower waste flow from higher recycling rates.

What is still redacted?

The vast majority of financial sums are redacted from the document, with with the authority invoking ‘regulation 12(5)(e)’ of the Enviromental Information Regulations (EIRs).

There are also a number of redacted sentences, where the nature of the sentence and the goal of redaction is unclear.

Are these redactions justified?

It is notable that in the contract FOI request Information Tribunal Ruling, for similar information in Annexe 4, the Tribunal stated (§77):

“No particular case is made as to how the redacted information in Annex 4 comes within regulation 12(5)(e) but, even assuming it did, we are satisfied the Commissioner’s assessment on the public interest is correct.”

However, they do ground some of this reasoning in the length of time between the information and the present day, stating:

Given that by April 2015 the Contract had long since been signed and there was controversy surrounding it we consider that there was a strong public interest in disclosure of all this detail. The Council’s Checklist says that release would have harmed its negotiating position, presumably in relation to a new procurement. We have commented on that scenario in general terms. Any information about the Council’s general financial position reflected in Annex 4 ought we think to have been in the public domain in any event.

This same reasoning would appear to apply in 2017 to figures from 2015.

The redactions in this report also cover ‘key changes in UBB proposal compared to the position at financial close’, including updated tonnage costs.

The World Bank Framework for Public Private Partnership disclosure calls for publication of tariff information, and revisions to tariff information: suggesting international best practice is for this information to be in the public domain, not kept confidential.

Where next?

Campaigners and County Councillors from a number of parties continue to oppose the incinerator. A complaint has been lodged with the Competition and Markets Authority and local residents have filed formal complaints with the Council’s monitoring officer over the conduct of Cabinet members reporting figures, largely it would seem based on the Ernst and Young report. There will undoubtedly be further updates in local press.

Now open access: The Daily Shaping of State Transparency: Standards, Machine-Readability and the Configuration of Open Government Data Policies

[Summary: co-authored #opendata paper now available open access]

A while back, Sam Goeta kindly invited me to collaborate with him on a paper around open data standards, and the work involved in the back rooms of open data. The paper was finally published in a special issue of Science and Technology Studies on Knowledge Infrastructures late last year, and the open access version is now available.

Abstract

“While many governments are now committed to release Open Government Data under non-proprietary standardized formats, less attention has been given to the actual consequences of these standards for knowledge workers. Unpacking the history of three open data standards (CSV, GTFS, IATI), this paper shows what is actually happening when these standards are enacted in the work practices of bureaucracies. It is built on participant-observer enquiry and interviews focussed on the back rooms of open data, and looking specifically at the invisible work necessary to construct open datasets. It shows that the adoption of open standards is increasingly becoming an indicator of the advancement of open data programmes. Enacting open standards involves much more than simple technical operations, it operates a quiet and localised transformation of bureaucracies, in which the decisions of data workers have substantive consequences for how the open government data and transparency agendas are performed.”

Javelin Park: What’s in the Information Tribunal ruling?

[Summary: exploring on a local open contracting campaign victory and it’s implications for contract transparency]

On Friday, the Information Rights Tribunal ruled on the appeal by Gloucestershire County Council against an earlier ruling by the Information Commissioner that the contract for a large PFI (Public Private Partnership) project to build an waste incinerator at Javelin Park near Stroud should be substantially published.

Campaigners have been fighting for access to the contract since 2015, when their first Freedom of Information Request was refused. Although we discovered earlier this year that the contract text had been accidentally put into the public domain by the Council failing to properly apply all the redactions they have been arguing for to an earlier FOI response, the Information Tribunal ruling is important in that it:

  • Sets out clearly the Tribunal’s view on the sections of the contract and it’s schedules that should be in the public domain (almost all of it);
  • Sets out clear reasoning applicable to other UK contracts – supporting the idea that there is a strong public interest in the text of contracts, and that exceptions for commercial confidentiality should be minimal;
  • Provides support for the idea that contract text should be proactively published.

You can find a copy of the ruling here, but, as it runs to 67 pages I’ve pulled out a few of the key points below.

(A) The basics

In paragraph 6 – 21 the Tribunal helpfully describe the background of the case – which involves a 25-year Public Private Partnership contract involving the build and operation of a Waste Incinerator, with an estimated overall contract value of £500m, and annual capacity of up to 190,000 tonnes.

The original request for an unredacted copy of the contract was made under the Environmental Information Regulations (EIR) – and was fought by the council on the grounds of Intellectual Property Rights, and Commercial Confidentiality.

(B) The arguments

Below is a non-exhaustive summary of arguments explored in the tribunal report (from the perspective of a non-lawyer trying to sense-make):

(1) Environment Information Regulations vs FOI? The council argued that sections of the contract should be considered under FOI (slightly weaker access rights) instead of Environment Information Regulations. The Tribunal ruled that the contract, as a whole, fell under EIRs (Para 39 & 40) as it, as a whole, represents a measure with substantial environmental implications.

(2) Commercial confidentiality? The council argued that large sections of the contract, including pricing and volume information, were commercially sensitive and their disclosure could pose a risk to both the private contractor, and the council, in terms of impacts on any future tendering activity.

In paragraph 44 the tribunal provide a useful discussion of EIR Regulation 12(5)(e) and the difficulty of working out to what extent an adverse effect of disclosure on economic interests of parties need to be established to justify confidentiality. However, the arguments of the Tribunal hinge much more on Schedule 23 of the contract itself, which was headed “Commercial sensitive contractual provisions”, and which was cited in the contract (§84.1) as the list of items that should be kept confidential by the parties.

A large quantity of the redactions sought by the Council, and which they appears to have spent over £200,000 fighting for (based on transactions to their lawyers in the Spending over £500 data), are not contained in this schedule.

Whilst it therefore appears the contract did follow good practice of agreeing from the outset any sections that could be subject to confidentiality, the Council did not follow this in actually applying redactions.

(3) Public interests in disclosure? The Tribunal evaluated each of the redactions sought by the council, and tested (a) whether confidentiality could be reasonably expected under the contract clause and schedules referring to this; and (b) whether there was, in any case, a public interest in disclosure.

Paragraphs 57 – 59 discuss the basis of public interest are worth quoting at length:

“§57. …Concerns have been expressed about the EfW technology chosen by the Council, which those against it say may involve harmful emissions and toxic waste left over from the scrubbing process. Planning concerns have been expressed about the height, mass and design of the plant and the increase in heavy road traffic which will be caused along with consequential air pollution. Although we are not in any position to assess the merits of these concerns, they are clearly genuine and not frivolous.

§58. The Contract itself is a PFI contract involving the expenditure of a great deal of public money over many years;… We can, we think, take judicial note of the fact that the PFI model is itself controversial, with legitimate concerns expressed about bad value for money, opacity and the tendency to load expenditure on future generations. Further, it is said that the structure of the Contract, by requiring the Council to pay for a certain amount of waste to be incinerated (the so-called “take or pay” arrangement) may have tied the Council in to supply a quantity of waste which is not viable in future and may have negative environmental effect of discouraging recycling…

§59. Given those considerations, in our view there was a significant public interest in the disclosure of the entire contract, in the interests both of transparency and accountability, ie the enable the public to be informed as to exactly what the Council had agreed on their behalf and its long-term consequences and to hold it properly to account, in particular through Council elections.”

On the issue of whether sections of the contract can be selectively disclosed, the Tribunal state:

§59. “… We make clear that we are not suggesting that the exercise is an ‘all or nothing’ one all we are doing is recognising that the provisions which the Council seeks to withhold are part of a greater interlocking whole and must inevitably be seen in that context.”

They also draw attention to the Local Government Transparency Code 2014 and the presumption in there of proactive disclosure – something I cover in this post.

They further draw attention to the fact that, when the original request was made in March 2015:

“§61…”the controversy was particularly intense and there was a danger that the whole Contract would have to be terminated at a cost, according to the Council of up to £100 million. At that stage, in our view, the Council’s obligation to act transparency was particularly strong as was the public interest in the exact position in relation to the compensation payable in so far as the Contract contained relevant provisions.”

They also argue that what matters is not how much of the text of a contract is in the public domain (the council argued that 95% of the text was public from the 1000+ pages of documents), but the substantives of that text. The tribunal state:

“In our view, the fact that the public authority has disclosed some information in the past cannot be relevant to the issue of whether they should have disclosed more.”

On the majority of individual redactions evaluated, the Tribunal find the public interest overwhelmingly supports publication. Paragraphs 74 – 216 go through the contract redaction-by-redaction, schedule by schedule, providing the reasoning for each decision. Where redactions are upheld, this is down to their information being included in the schedule of confidential information, and the Tribunal finding no substantial public interest in disclosure (though in some cases they still express puzzlement as to why redaction might be required).

(4) Impact on future procurement? In paragraph 72 the Tribunal consider arguments from the Council and UBB that disclosure would prejudice future procurements, and prevent the Council getting the best deal. They state:

“§72… We cannot accept such a case. Any potential contractor seeking to do business throughout the EU must be well aware of the duties of public authorities in relation to environmental information. We do not accept that they would (or should) complain or change their behaviour in response to a disclosure of information by the Council or any other public authority which was required by the EIR (or indeed FOIA)…”

(5) Intellectual property protection? The council invoked a separate argument for Schedule 33 which covers the sale of electricity generated from the plant. The mechanism by which this is to happen is fairly opaque, and appears to involve as-yet untested deals for ‘power off-take’. The Tribunal note that they were not “…given a very clear explanation of how this was all going to work…” (§212), but that “…Mr Mawdsley [(the council officer responsible)] [hoped] to sell the contents of Schedule 33 to other local authorities.”

The council argued that the Schedule was their IP, “based on copyright , database rights and the law relating to trade secrets.”.

The tribunal dismiss this, and in a damning paragraph note:

“§216. So far as the public interest is concerned, we agree with the Commissioner that, if relevant, it favours the disclosure of Schedule 33. The Council expressly accepts that there is a public interest in transparency about its plans to sell electricity for wider use; in our view it is a weighty public interest. On the other hand, we remain unclear as to how the Council’s or UBB’s negotiating position with third parties will be damaged. As to the wish to protect the confidentiality of legal and technical details that are novel in order to sell them on to other local authorities, even assuming that Mr Mawdsley is not being overoptimistic about the potential for the Council to make money in this way, we do not think that there is a particularly great public interest in the Council being able to commercially exploit a scheme which is apparently designed to avoid the normal regulatory regime.”

Overall

I’m not sure to what extend Tribunal decisions set precedents for others – but it seems to me there are strong arguments here that supports the positions that:

  • Where contracts are made that commit public money – the public have a right to know the detail of those contracts;
  • Contracts need to be treated as a whole, and redactions kept to a minimum;
  • Only redactions agreed in advance, and set out transparently in a clear schedule should be allowed;
  • A public party cannot claim intellectual property over a negotiated contract text;

Now that we have official access to the substantial majority of the Gloucestershire Incinerator contract, the challenge ahead is to work our what of the damage done by the Council Cabinet and contractors unaccountable actions over the last 18 months can be challenged, and undone. Access to documents is ultimately just one part of a wider open contracting journey.

(C) Other things of note

There are a few other elements of note I jotted down whilst reading through the judgement.

  • The claim made to council on 18th Feb 2015 that it could cost £60m – £100m to cancel the contract appears to be based on calculations from officers, and/or Ernst and Young which have not been published by the authority (perhaps another EIR or FOIA request will be needed here…). The Tribunal ruling refers in Paragraph 27 to a document from Ernst and Young presented to Cabinet in November 2015. However campaigners reading the unredacted contract cannot find the substantiation for the cancellation costs being so high before the facility is operational. It appears breakage before the plant is in operation could cost substantially less than the break-points once it is up and running – and possibly even lower than the £30m the Council has subsequently committed from reserves to cover shortfalls in the project.

  • Fighting disclosure has potentially cost the council much more than the hundreds of thousands spent on legal fees. Now that the contract model can be scrutinised, and alternatives explored, it may turn out that delays have led to potential cancellation of the contract

  • Mr Mawdsley, the council officer who has been pushing the Incinerator contract, comes in for criticism from the Tribunal. In paragraph 73 they note “Mr Peiro’s [UBB staff member] evidence was inevitably likely to be rather partisan, and, although he is an official and was giving evidence on behalf of a public authority, we are afraid we reached the view that Mr Mawdsley’s evidence on behalf of the Council was also rather partisan. We were surprised at the failure of each to attach any great importance to clause 84 and Schedule 23 or to the Transparency Code… We accept the submission of Mr West at para 78 of his final submissions that Mr Mawdsley’s evidence was ‘…so far-reaching as to be unconvincing, in particular in relation to matters such as Access Road Disruption Events”.

What next?

I’ve tried, at least in section A and B above to summarise rather than analyse. But, as I’m posting this on a personal blog, if I might here be forgiven a personal and partisan point…

If you are in Gloucestershire and concerned about this – the Tribunal made a good point: elections are a key mechanism to hold the Council to account – and all the Councillors who voted for the Incinerator on the basis of bad information, or secrets known only to Cabinet, are up for re-election on May 4th.

I only discovered this local case when my wife, Rachel, started getting involved in local Green Party conversations, and pointed out the work our Green County Councillor Sarah Lunnon was doing to push for open contracting, and to challenge the secrecy of the Incinerator contract. We were both astonished to see the County Council being so reckless with public resources and our local environment – and to see them so opposed to transparent and accountable politics.

It spurred me into reading as much as I could of the information that was available on the contract – but Rachel has taken it a step further – and is standing as Green Party candidate for Minchinhampton Division in the upcoming County Council elections.

There is ultimately the chance that we could change the balance of power in Gloucestershire – voting out the Tory administration that’s made these reckless decisions – and getting in a progressive coalition who can work to undo the damage. So – if you happen to be local to Minchinhampton, Thrupp or Chalford: please support Rachel. If you live elsewhere in Gloucestershire: make sure you get out and vote on May 4th, and use your vote for a progressive candidate who will commit to open contracting, and to stopping this one wasteful incinerator deal.

And if this is all too parochial… think about the contracts your local authority has signed you up to. Have you looked to see that they really work in the public interest?


[Note – typos in transcription from the original judgement are my own. I’m working from a printed copy, awaiting access to digital copy from the Tribunal website]

UK Open Contracting goes local in Gloucestershire?

[Summary: Explore arguments for Gloucestershire County Council to support Open Contracting on Weds 7th December]

This Wednesday, on the eve of the Open Government Partnership summit in Paris, where I expect we’ll be hearing updates from Open Contracting projects across the world, my local County Council in Gloucestershire will be voting on an Open Contracting motion.

screen-shot-2016-12-04-at-15-29-45

If you’re not familiar with Open Contracting, it’s a simple idea, described by the Open Contracting Partnership here.

The motion, introduced by our local Green Party councillor Sarah Lunnon, calls on the Council to:

commit to the Open Contracting Global Principles and takes action to ensure that: – By the end of 2017 complete information for all contracting processes over £1m, including details of the tender, award and contract process, the full text of contracts and amendments, and performance information, are proactively published;* – By the end of 2018 complete information is available for all contracting processes, over £500, is proactively published”

It goes on to state that:

“The presumption should be that the text of all contracts is open by default. Redactions should only be permitted: (a) at the explicit written request of the parties to the contract; (b) subject to the public interest tests of the Freedom of Information act; (c) with the minimum possible redactions; and (d) with full justifications for any redaction given.”

Passing this motion would, as far as I’m aware, make Gloucestershire the first local authority in the UK to explicitly commit to the Open Contracting Global Principles. Although the Gloucestershire motion may be, at least in part, a response to the opaqueness of one particular contract, by being framed in terms of the Global Open Contracting Principles, if offers something of a win-win for local business (greater access to opportunities), citizens (greater understanding of how funds are spent) and the authority (better deals, and better scrutiny of spending).

screen-shot-2016-12-04-at-15-31-49

If you’re a Gloucestershire resident, consider contacting your Councillor to ask them to support the motion.

If you’re not – perhaps there might be an opportunity to bring forward an open contracting initiative in your own area? As it turns out – all the national policy foundations are in place in the UK – it just needs commitment from local areas to put them into practice.

The framework exists for local open contracting in the UK

Below are a few of the resources I found to address common questions raised about Open Contracting, and disclosure of contracting documents, when I was researching a short letter to local councillors on the motion.

National procurement policy already establishes a presumption in favour of disclosure The UK’s Public Sector Procurement Policy incorporates a set of Transparency Principles that state that:

There should be a presumption in favour of disclosing information, with exemptions following the provisions of the Freedom of Information Act – for example, on national security or commercial confidentiality grounds. The presumption in favour of disclosure should apply to the vast majority of commercial information about government contracts, with commercial confidentiality being the exception rather than the rule.

The principles go on to note that exemptions may be available for pricing information, but that “This means the way the supplier has arrived at the price they are charging government in a contract, ****but should not usually be grounds for withholding the price itself.****” (emphasis added)

Commercial confidentiality concerns can be addressed through good planning

In 2014, the Centre for Global Development facilitated a multi-stakeholder working group on Publishing Government Contracts, including public and private sector representatives. This group concluded that:

While there are legitimate commercial, national-security, and privacy concerns, they involve a small minority of contracts and can be addressed using a principles-based redaction policy. (From issue brief. Full report here)

The findings of the CDG working group show that a principle of ‘open by default’ is viable and practice.

Data protection concerns rarely justify non-disclosure of contracts

The Local Government Transparency Code 2015 provides guidance on managing any data protection concerns that may arise from contract publication, stating that:

The Data Protection Act 1998 also does not automatically prohibit information being published naming the suppliers with whom the authority has contracts, including sole traders, because of the public interest in accountability and transparency in the spending of public money.

Section 20 of the code addresses commercial confidentiality, stating that:

The Government has not seen any evidence that publishing details about contracts entered into by local authorities would prejudice procurement exercises or the interests of commercial organisations, or breach commercial confidentiality unless specific confidentiality clauses are included in contracts. Local authorities should expect to publish details of contracts newly entered into – commercial confidentiality should not, in itself, be a reason for local authorities to not follow the provisions of this Code. Therefore, local authorities should consider inserting clauses in new contracts allowing for the disclosure of data in compliance with this Code.

Model transparency clauses can be used to manage disclosure of structured performance information

Initial advocacy for a Model Transparency Clause in the UK was led by Institute for Government, and the national transparency clause, now included in the Model Services Contract, was developed with input from the National Council for Voluntary Services, Open Data Institute, and major private sector contractors. The extensive work that has taken place to develop models of disclosure that balance commercial practicalities and government and public interests in having access to clear information on contract performance, provides a tried-and-tested template for local authorities to build upon.

Contracts Finder and the Open Contracting Data Standard provide ready-made tools to implement disclosure

Contracts Finder is the government’s national platform for publication of contracting opportunities and awards. Local authorities are mandated to submit above-threshold procurements through the platform, but, as far as I understand, can also submit data on all procurement opportunities via Contracts Finder if they choose.

This means there is little extra cost for a local authority to make structured information on all its procurement processes available.

Some planning may be required to manage contract document publication effectively, but the technical complexity involved should be no more than making space available on a local council website for the documents, and linking to these from submissions to Contracts Finder.

Contracts Finder has recently launched an Open Contracting Data Standard API, allowing access to structured information about contracting processes, and a discovery phase is currently underway to improve the platform: with the opportunity to feed in ideas about how local authorities might wish to get their data back, to be able to display it for contracting transparency locally.

Looking ahead

I hope I’ll be able to update this post with news of a successful motion on Thursday. In any case, I’ve been quite struck when working on the research above on the potential to really develop a local open contracting agenda in the UK.

Interested in getting involved too? Drop me a line and let’s explore (tim@timdavies.org.uk)

ConDatos Talk notes: Open data as strategy

[Summary: Notes from a conference talk]

Last week I was in Colombia for AbreLatam and ConDatos, Latin America’s open data conference. Thanks to a kind invitation from [Fabrizio Scrollini](http://www.twitter.com/Fscrollini], I had the opportunity to share a few thoughts in one of the closing keynotes. Here is a lightly edited version of my speaker notes, and full slides are available here.

Open Data as strategy

screen-shot-2016-11-08-at-18-05-11In a few months, Barak Obama will leave the White House. As one of his first acts as US President, was to issue a memorandum on Transparency and Open Government, framed in terms of Transparency, Participation and Collaboration.

This memo has often been cited as the starting gun for a substantial open data movement around the world, although the roots of the open data movement go deeper, and in many countries adopting open data policies, they have blended with long-standing political priorities and agendas.

For myself, I started studying the open data field in 2009: exploring the interaction between open data and democracy, and I’ve been interested ever since in exploring the opportunities and challenges of using open data as a tool for social change.

So, it seems like a good time to be looking back and asking where have we got to eight years on from Obama’s memo, and nearly ten years since the Sebastapol Open Government Data Principles?

We’ve got an increasing number of datasets published by governments. Data portals abound. And there are many people now working in roles that involve creating, mediating, or using open data. But we’ve still got an impact gap. Many of the anticipated gains from open data, in terms of both innovation and accountability, appear not to have been realised. And as studies such as the Open Data Barometer have shown, many open data policies have a narrow scope, trying to focus on data for innovation, without touching upon data for transparency, participation or collaboration.

Eight years on – many are questioning the open data hype. We increasingly hear the question: with millions of datasets out there, who is using the data?

My argument is that we’ve spent too much time thinking about open data as an object, and not enough thinking about it as an approach: a strategy for problem solving.

Open data as an approach

screen-shot-2016-11-08-at-18-07-39
What do I mean by this?

Well, if you think of open data as an object, it has a technical definition. It is a dataset which is machine-readable, published online, and free to re-use.

The trouble with thinking of open data in this way is that it ignores the contents of the data. It imagines that geodata maps of an entire country, when published as open data, are the same sort of thing as hospital statistics, or meta-data on public contracts. It strips these datasets from their context, and centralises them as static resources uploaded onto open data portals.

But, if we think of open data as an approach, we can get towards a much clearer picture of the kinds of strategies needed to secure an impact from each and every dataset.

What is the open data approach?

Well – it is about transparency, participation and collaboration.

So many of the policy or business problems we face today need access to data. A closed approach goes out and gathers that data inside the organisation. It uses the data to address the local problem, and all other potential uses of that data are ignored.

An open approach considers the opportunities to build shared open data infrastructure. An infrastructure is not just something technical: it involves processes of governance, of data quality assurance, and community building.

Building an open infrastructure involves thinking about your needs, and then also considering the needs of other potential data users – and working together to create and maintain datasets that meet shared goals.

Ultimately, it recognises open data as a public good.

Let me give an example

In the United States, ‘211 Community Directory services play an important role in helping refer people to sources of support for health or social welfare issues. Local 211 providers need to gather and keep up-to-date information on the services available in their area. This can be expensive and time consuming, and often involves groups collecting overlapping information – duplicating efforts.

The Open Referral initiative is working to encourage directory providers to publish their directories as open data, and to adopt a common standard for publishing the data. The lead organiser of the initiative, Greg Bloom, has invested time in working with existing system vendors and information providers, to understand how an open approach can strengthen, rather than undermine their business models.

In the early stages, and over the short-term, for any individual referal provider, getting involved in a collaborative open data effort, might involve more costs than benefits. But the more data that is provided, the more network effects kick in, and the greater the public good, and private value, that is generated.

This demonstrates open data as an approach. There isn’t an open referral dataset to begin with: just issolated proprietary directories. But through participation and collaboration, groups can come together to build shared open data that enables them all to do their job better.

It’s not just about governments pubishing data

It is important to note that an open data approach is not just about governent data. It can also be about data from the voluntary sector and the private sector.

With targetted transparency policies governemnts can mandate private sector data sharing to support consumer choice, and create a level playing field amongst firms.

As in the Open Referral example, voluntary sector and government organisations can share data together to enable better cross-organisation collaboration.

One of the most interesting findings from work of the Open Data in Developing Countries research network in Brazil, was that work on open data created a space for conversations between government and civil society about processes of data collection and use. The impact of an open data approach was not just on the datasets made available, but also on the business processes inside government. By engaging with external data re-users, government had the opportunity to rethink the data it collected, with potential impacts on the data available for internal decision making, as well as external re-use. We are seeing the same thing happen in our work on Open Contracting, which I will discuss more shortly.

The falacy of more data now

Before I move on, however, I want to caution against the common ‘falacy of more data now’.

There are many people who got into working with open data because they care about a particular problem: from local transport or environmental sustainability, to accountable politics, or better education. In exploring those problem, they have identified a need for data and have allied with the open data movement to get hold of datasets. But it is easy at this point to lose sight of the original problem – and to focus on getting access to more data. Just like research papers that conclude calling for more research, an open data approach can get stuck in always looking for more data.

It is important to regularly loop back to problem solving: using the data we do have available to address problems. Checking what role data really plays in the solution, and thinking about the other elements it sits alongside. Any only with a practical understanding, developed from trying to use data, of the remaining gaps, iterating back to further advocacy and action to improve data supply.

Being strategic

screen-shot-2016-11-08-at-18-07-49
So, if open data is, as I’ve argued, an approach, how do we approach it strategically? And how do we get beyond local pilots, to impacts at scale?

Firstly, ‘open by default’ is a good starting point. Strategically speaking. If the default when a dataset is created is to share it, and only restrict access when there is a privacy case, or strong business case, for doing so – then it is much easier for initiatives that might use data for problem solving to get started.

But, ‘open by default’ is not enough. We need to think about standards, governance, and the ecosystem of different actors involved in creating, using, providing access to, and adding value on top of open data. And we need to recognise that each dataset involves it’s own politics and power dynamics.

Let’s use a case study of Open Contracting to explore this more. Colombia has been an Open Contracting leader, one of the founder members of the Open Contracting Partnership, and part of the C5 along with Mexico, France, Ukraine and the UK. In fact, it’s worth noting that Latin America has been a real leader in Open Contracting – with leading work also in Paraguay, and emerging activities in Argentina.

Open Contracting in focus

Public contracting is a major problem space. $9.5tn a year are spent through public contracts, yet some estimates find as much as 30% of that might leak out of the system without leading to public benefit. Not only can poorly managed contracts lead to the loss of taxpayers money, but corruption and mismanagement can be a source of conflict and instability. For countries experiencing political change, or engaged in post-conflict reconstruction, this issue is in even sharper relief. In part this explains why Ukraine has been such an Open Contracting leader, seeking to challenge a political history of corruption through new transparent systems.

Open Contracting aims to bring about better public contracting through transparency and participation.

Standards

To support implementation of open contracting principles, the Open Contracting Partnership (OCP) led the development of OCDS – the Open Contracting Data Standard (OCDS). When working with the Web Foundation I was involved in the design of the standard, and now my team at Open Data Services Co-operative continue to develop and support the standard for OCP.

OCDS sets out a common technical specification for providing data on all stages of a contracting process, and for linking out to contracting documents. It describes what to publish, and how to publish it.

The standard helps Open Contracting scale in two ways:

  • Firstly, it makes it easier for data publishers to follow good practices in making their data re-usable. The standard itself is produced through an open and collaborative process, and so someone adopting the standard can take advantage of all the thinking that has gone into how to model contracting processes, and manage complex issues like changes over time, or uniquely identifying organisations.

  • Secondly, the standard is built around a number the needs of a number of different users: from the SME looking for business opportunities, to the government official looking to understand their own spend, and the civil society procurement monitor tracking contract delivery. By acting as a resource for all these different stakeholders, they can jointly advocate for OCDS data, rather than working separately on securing separate access to the particular data points they individually care about.

Importantly though, the standard is responsive to local needs. In Mexico, where both the federal government and Mexico City have been leading adopters, work has taken place to translate the standard, and then to look at how it can be extended and localised to fit with national law, whilst also maintaining connections with data shared in other countries.

Governance & dialogue

When it comes to enabling collaboration through open data, governance becomes vitally important. No-one is going to build their business or organisation on top of a government dataset if they don’t trust that the dataset will be available next year, and the year after that.

And governments are not going to be able to guarantee that they will provide a dataset year after year unless they have good information governance in place. We’ve seen a number of cases where data publishers have had to withdraw datasets because they did not think carefully about privacy issues when preparing the data for release.

For Open Contracting, the standard itself has an open governance process. And in Open Contracting Partnership ‘Showcase and Learning Projects’ there is a strong emphasis on building local partnerships, making sure there is dialogue between data publishers and users – creating the feedback loops needed to build a data infrastructure that can be relied upon.

In the UK, adopting of OCDS will soon give the government a better view of how far different departments and agencies are meeting their obligations to publish contracting information. By being transparent with the data, and being transparent about data quality, civil society and the private sector can get more involved in pushing for policies to be properly implemented: combining top-down and bottom-up pressure for change.

Support and community

One of the most important lessons for us from Open Contracting has been that scaling up open data initiatives is not just about standards and technical specs, but it is also about relationships, community and providing the right support at the right time.

The Open Contracting Partnership invest in bringing together champions of open contracting from across the world to get inspired and to share learning. Because they are working with common standards, ideas and tools are more easily transferable. And as I mentioned earlier, thinking about how to improve their open data also creates opportunities for groups to think about improving their internal systems and processes.

In addition, my team at Open Data Services Co-operative provide the ‘technical helpdesk’ for OCDS. We offer e-mail, phone and workshop support to governments working to publish their data, and to groups seeking to use open contracting data. Our goal is to make sure that when data is published, it is easy-to-use, and that all the small barriers to data re-use that exist for so many other datasets are not there when you come to an open contracting dataset.

We do this because data standards are only as strong as their early implementations. But we’re not aiming to be the only support provider for OCDS. In fact, we’re aiming to stimulate an ecosystem of support and data re-use.

Ecosystem

A strategic approach to problem solving with open data needs us to recognise the different roles in a data value chain. And to think about what elements need to be kept open for a vibrant ecosystem, and where to create space for proprietary business models.

If governments need consultancy support to improve their systems to produce OCDS data, and a marketplace of expert consultants develops, this is a good thing for scaling adoption. If private firms build value-added data analysis tools on top of contracting data, this is something to welcome that can scale use.

But if the originally published data is messy, and firms have to spend lots of money cleaning up the raw data before they use it, then barriers to market entry are created. This stiffles innovation, and leads to services only accessible to wealthy private sector, excluding civil society data users.

That’s why there is a need for a range of different actors, public, civil society and private, involved in a data ecosytem – and space for a range of business models.

Business models

I’ve been asked to touch in particular on business models in this talk – not least because the company I’m part of, Open Data Services Co-operative, has been exploring a different model to scale up support for open data.

We’re set up as a Workers Co-operative: a model where Latin America has a strong history. In a workers co-op, the staff own the business: and make decisions about it’s future. This might not sound that significant, but it has a number of distinction against other models:

(1) It’s a business, not a charity. This can give us the flexibility to innovate, and the drive to find sustainable models for our work. Right now, we work through a mix of contracts for technology research and development, and through providing ongoing support for data standards, often ultimately funded by donors who believe in investing in public good open data infrastructure.

(2) Organic instead of investment growth. A lot of the examples used when talking about tech businesses are born out of massive-scale silicon valley investments. Most co-operative models are based on growing through member contributions and revenue, rather than selling equity. Although we are set up as a workers co-op, there are growing discussions around ‘platform co-operatives’ and ‘data co-operatives’, in which those who could benefit from shared data infrastructure collectively support its development through a co-op model.

(3) Social mission focus. We want to provide jobs for people – particularly growing the community of professionals working on open data, as we recognise there are limited opportunities for stable open-data and social change focussed jobs. But we also want to have an impact on the world, through enabling open data-approaches to problem solving. As a worker owned business, we’re not focussed on profit for shareholders or an external owner, but on developing effective projects, and contributing to the wider community and issues we care about.

When it comes to scale, for a co-operative the question is about reaching the right scale, not about unlimited growth. That’s why as demand has been growing for support on the Open Contracting Data Standard in Latin America, we’ve been working with the Open Contracting Partnership to put out a call for a new partner organisation to take on that role – co-operating alongside Open Data Services to provide services across the region.

If anyone would like to find out more about that opportunity – please do check out the details online here.

I’m not here to argue that co-operatives are the only or the best business model for working with open data – but I do encourage you to think about the different models: from supporting individual entrepreneurs, to building open data activities into existing organisations, and supporting the emergence of co-operative groups that can catalyse a wider re-use ecosystem.

Recap

So let’s recap.

  • Open data is an approach, not an object

  • Open approaches win out over closed approaches when it comes to creating both social value, and opportunities for innovation and growth

  • But, we need to be strategic about open data: using it for problem solving, and making sure data quality and reliability is good enough for ongoing re-use

  • An we need sector-specific approaches, with a mix of different organisations involved

I’ve shared case studies of Open Referral and Open Contracting. But in other sectors the right approaches may be different.

My challenge to you is to think about you can apply values of transparency, participation and collaboration to your open data projects, and how you can act strategically to make use of standards, community engagement and support for data publishers and users in order to build vibrant ecosystems.

Open Government – Gouvernement Ouvert: Same same but different

[Summary: preliminary notes for a roundtable discussion on open government research]

I’m talking tomorrow at a workshop in Paris to explore the research agenda on Open Government. The first panel, under the title “Open Government – Gouvernement Ouvert: Same same but different?” aims to dig into the question of whether open government takes different forms in different countries and contexts.

To what extent is open government about countries moving towards some set of universal principles and practices for modern accountable and participatory governance? And to what extent is it about diverse culturally specific reforms to existing national systems of governance? We’ll be getting into these questions by looking at the open government situation in a number of different (European) countries, followed by a roundtable discussion.

The organisers have set three questions to get discussions going. I’ve jotted down the thoughts below by way of my preparation (and sharing here in the spirit of blogging things before I try and edit them too much, which means they never make it out).

Question 1: What has been done lately in the UK that could qualify as open government?

(1) Brexit and open government

It’s hard to answer this question without first considering the recent EU Referendum, and subsequent political fall-out of the Brexit vote. How does this fit into the open government landscape?

In general, democratic process, elections and referenda have fallen outside the scope of an open government agenda. These votes might be the means by which we choose the legislative branch of government, and through which mass civic input is solicited, but when it comes to open government discourse, focus has been placed firmly on the executive and administrative branch. Whether this is sustainable in future is an open question (indeed, the OGP is moving towards a greater engagement with legislatures and legislative process).

An analysis of the EU Referendum, even though it engaged more voters than the last general election in directly addressing a substantive policy issue , would find it to be far from a model of open government. The abuse of statistics by all sides during the campaign, and the lack of attention given to substantive debate, represent failures of both political integrity from campaigners, and a failure of effective scrutiny from the media.

The subsequent position of the new administration, interpreting the referendum vote without any process of dialogue with the parliament, let along the wider public, demonstrates a retreat from ideas of open government, rather than an engagement with them. Rather than addressing social divisions through dialogue, government appears to be pursuing policies that deepen them.

At the same time it is worth noting how success and failures of open government may have contributed to the referendum result. The Open Government declaration talks of harnessing new “technologies to make more information public in ways that enable people to both understand what their governments do and to influence decisions.”. When it comes to British citizens understanding the EU, it is clear that much of the information that was available was not making it through, and few felt about to influence decisions at this supranational level. However, it is also clear that where data was available, on budgets, spending, regulation and more – that information alone was not enough to lead to better informed citizens, and that simply adding data does not make for more informed debate, or more open governance.

This raises some big questions for open governance advocacy in the UK: whether future action should engage with bigger political questions of rights, representation, media ethics and political integrity? Or whether these issues are part of a separate set of agendas to revisit, rethink and revitalise our democratic systems: whilst open government should remain focussed on administrative reforms for a more efficient, effective and responsive state?

(To explore answers to these questions I would argue there is much UK open government advocates can gain from approaching the Open Government Partnership as a space to learn from countries where the rights, freedoms and values we have often taken for granted are only recently won, or are still being fought for.)

(2) From open data to data instructures

When we look at the explicit open government commitments of the UK in most recent OGP National Action Plan, it is clear that the focus is firmly on the administrative side of open government. And very much on data and technology.

Of the 13 commitments, 8 are explicitly about data – representing the strong bias towards open data that has been present throughout the UK’s engagement in the Open Government Partnership. Because the most recent National Action Plan was published at the UK Anti-Corruption Summit in May, there is also a strong emphasis on data for anti-corruption. Asides from a process commitment to ongoing dialogue with civil society in developing the action plan itself, and a focussed set of engagement plans around how private sector and civil society actors should be involved in shaping the data that government publishes, there is little in the latest NAP on participation.

What is interesting to note however, is the move away from general commitments on open data, to a much more explicit focus on specific datasets, and the idea of building data infrastructures. The commitments cover publishing data on beneficial ownership for companies bidding on UK government contracts or owning property in the UK, gathering more structured extractives industry reporting, adopting the open contracting data standard for procurement data, publishing government grants data using the 360 Giving standard, and working towards standardised elections data. I’ll return shortly to the global nature of these commitments, and the infrastructure being constructed.

Effectively implemented, disclosure of this data will qualify as open governent on the ‘output side’. However, the challenge remains to articulate in future versions of the National Action Plan the ‘input’ side for these initiatives. For example, we are, as yet, to articulate in the NAP the feedback loops through which, for example, a commitment to Open Contracting can be made not just about publishing data on contracts, but also about creating more opportunities and mechanisms for citizen engagement and oversight of contracting.

(3) Process and product

In a somewhat meta-step, the OGP National Action Plan itself is also often considered to be an interesting act of open government. Since the second NAP, there has been close engagement between officials and a civil society network to shape the plan. The plan itself was published with joint forewords from the Minister for Cabinet Office, and the Civil Society Network. This kind of ‘open policy making’ process has been explored as a template for a number of other policy areas also, although with less concrete joint outputs.

Increasingly I’m reflecting on whether to date this process has found the right balance between government and civil society collaboration on core reforms, and the risk of civil society being co-opted: securing formal practices of transparency, but doing little to translate that into accountability.

When asked ‘What has been done lately in the UK that could qualify as open government?’, I would like to be able to answer with stories of civil society actions that use transparency to call government more to account – yet I’m struggling to identify such stories to share.

Question 2: What are the main open government issues in the UK and what are their political impacts?

There are three main trends I want to focus on in addressing this second question: privatisation and private sector accountability, anti-corruption and devolution.

Privatisation and private sector accountability

Firstly, privatisation. More and more we are seeing public services contracted out to the private sector. Instead of lines of management accountability from elected officials, through administrators, to front-line staff, the relationship between governments and fron-line services has become a contractual one. This changes structures of accountability and governance. If a service is not being delivered adequately, the levers which government can pull to fix it may be constrained by the terms of a contract. That makes opening up contracts a particularly important area of open government right now.

On a related note, it is worth noting that many reforms, such as open contracting, extractives industry transparency, beneficial ownership transparency, and the emerging area of tax transparency, are not solely about holding governments to account for the use of public funds, but also extend to scrutinising the actions of the private sector, and trying to rebalance power between citizens, state and private sector.

Anti-corruption

Secondly, as noted above, under Prime Minister Cameron, the UK Government placed a strong emphasis on the anti-corruption agenda, and on open government as a key mechanism in the fight against corruption. Whether the political will to prioritise this will continue under the new administration remains to be seen. However, a number of components of an anti-corruption data infrastructure are being put in place – albeit with major gaps when it comes to lobbying transparency, or structured data on interest and asset declarations.

Devolution

Thirdly, devolution. Although it might not be evident from the current UK OGP National Action Plan, many areas of the open government agenda are devolved responsibilities. By the end of the year we hope to see separate commitments in the NAP from Scotland, Wales and Northern Ireland. Scotland is one of the sub-national OGP pilot regions. And as more UK cities and regions get elected mayors, there is scope to build on the sub-national OGP model in future. However, with the regions of the UK controlled by different political parties, this raises interesting challenges for the open government agenda: whether it will lead to a politicisation of the agenda, or a further focus on depoliticised technical reforms is yet to be seen.

Question 3: Is there a specific UK perspective on open government?

Reading both forewords to the latest UK OGP National Action Plan, it appears to me that within the UK there are multiple perspectives on open government. Whilst the then Minister for Cabinet Office placed the empahsis on using “data to make decisions, and where a free society, free markets and the free flow of information all combine to drive our success in the 21st century”, the Civil Society forword talks of open government as a “building block for a more democratic, equal and sustainable society.”

However, when looked at alongside other OGP member nations, we can make a number of observations about UK angles on open government:

  1. The UK appears to be part of a cluster of technical advanced countries, who are making strong links between agendas for open government and agendas for technical reform inside the state.

  2. Civil Society advocacy on open government in the UK has been strongly influenced by international NGOs based in London/the UK, with a dual focus on the domestic reform, and the role of the UK as a key actor in global initiatives, such as IATI and the EITI. The government has also placed strong emphasis on international initiatives, such as beneficial ownership transparency and open contracting.

  3. This emphasis on international initiatives, and the recent link between the OGP National Action Plan and the May anti-corruption summit, has led to a particular focus on data standards and interoperability. This highlights the global component of open government: building data infrastructures that can be used to secure accountability in an era of highly mobile global finance, and in which sovereign states cannot fight corruption within their borders alone.

How this compares to the emphasis of initiatives in France, and the other countries to be considered on tomorrow panel is something I’m looking forward to exploring.