The Hazards of Open Data Exceptionalism

Frustrated users, courtesy naersjoen. Licensed CC-BY-NC-SA.

The prospect of funding cuts for e-Gov initiatives like, and friends is worrying. Everyone should join the Sunlight Foundation’s effort to Save the Data. At the same time, this is a good opportunity for reflection.

There’s no doubt that the proliferation of Open Government websites has been a great first step for transparency and accountability. Despite the flaws, most of us see the promise of something very powerful in these projects.

I can feel strongly about the value of these programs and still be mystified at the $18M cost of when RATB has surely already built their own internal system to do basically the same thing. This has me thinking.

Why create one set of tools for citizens, and another for internal use? It seems that services like should be part of the usual operation of OMB, rather than some special e-Gov project that’s vulnerable to budget cuts. Why a distinct and conspicuous line item for, when it’s a citizen-friendly face on the $24M Federal Procurement Data System? Why not spend that money instead on improving FPDS, and making it more usable for both the public and the government?

There’s an argument, I suppose, that the kind of tools an Inspector General needs should be different than those a curious citizen might need. That’s probably true. It’s probably also true that if the tools are inscrutable to the unskilled citizen, they’re inscrutable for the government expert, as well. Interface designers handle different users with different comfort levels all the time. It is perfectly possible to build a system that’s simultaneously useful to the beginner and the expert.

By bringing citizens closer to the tools and data that are already in place, we can reduce the pernicious data translation and latency problems, spend less money on redundant visualizations, spend more money on improving the decision-making tools, and make Open Government a permanent part of the Federal IT infrastructure.

2 thoughts on “The Hazards of Open Data Exceptionalism

  1. Well said. I also think many of us can be ardent supporters of these efforts and still a bit mystified at the budgets required to keep them running. Perhaps it’s due to the gauntlet of the federal contracting process, and the particular demands drove a lengthy and feature-rich development process. But one is tempted to say that if the tool/platform development had been conducted openly and resulted in reusable open source code, then the development costs would be lower and the costs to operate likely lower as well. What large company, NGO, or government would not want their own or IT Dashboard? It would also mean that a re-compete for the contract to operate a site like could be much more competitive. It’s great that the IT Dashboard was finally released, but it’ll take awhile before it achieves the kind of velocity and leverage that it could have achieved had it been open from day one.

    Open data advocates need to argue as vociferously for open source code and dev processes as they do for open APIs and data sets. The latter is not long-term resilient without the former.



  2. I think you’re absolutely right that there’s fat to be trimmed from these efforts. After launch, in particular, it should be relatively cheap to maintain many of these sites on an ongoing basis. The data, storage and traffic needs should typically put maintenance efforts for a given site in the tens of thousands or low hundreds of thousands of dollars per year range — not millions.

    Of course, this isn’t an ideal time to be addressing these issues. Finding efficiencies while leaving room for experimentation is important — the equivalent of e-Gov projects committing to lose weight and get in shape. But the way to achieve that is clearly not by saying “we agree you should lose 50 lbs; you can pick what to cut off” — which is what the originally proposed budget reduction amounted to.

    Finally, I agree that going to the root of these systems is important: it’s the only way to address some of the fundamental pathologies like poor data quality that make them less than fully useful. But I think it’s a mistake to abandon integrated efforts. FPDS-NG is maintained by incumbent agencies and has its own points of brittleness and ossification. There can be real benefits to integration programs that are consumer-facing, and which work at a higher level of abstraction. It’s a chance to try to reconcile disparate data models (e.g. grants and contracts) and to push the ball forward in ways that the old guard wouldn’t even consider (e.g. disclosure of contract language). And it’s a way to acknowledge that regulators, IGs and other oversight officials really do have different responsibilities and workflows than journalists and citizens.


Comments are closed.