An Agency that Learns

In the end I like structures that are human-shaped, not idea-shaped and humans are great heaps of inconsistency, ambiguity and complexity. All I’m saying is that if you expect this to be a kind of Open Source madrassah you will be disappointed.

– Stephen Fry, “A notice to staff and parents, a little housekeeping…”

The GSA and its cabal of hipster-led innovation¹, 18F, both released remarkable new open source policies. Shortly thereafter, OMB announced the US Digital Service, and its Playbooks, which also encourage the use of open source. These policies embrace and extend the groundbreaking “everything we do is open” policy at the Consumer Financial Protection Bureau, formally making open source the default for new projects. You can read the GSA announcement, the 18F blog post, and the USDS Playbook for the details, but suffice to say that this is the culmination of years of work from advocates inside and outside of government, and it’s a triumph.

This is a milestone for the US Government, and it’s long overdue. You can’t run an organization today without knowing how to use open source. It’s where innovation happens. Yahoo released Hadoop, and launched the multi-billion dollar Big Data industry. Google released Android and changed mobile computing. Twitter promotes Mesos, a total rethinking of how you schedule workloads. dotCloud released Docker, and stands to permanently change how we deliver applications. The unlikely crew of Google, Microsoft, and Red Hat are all working on Kubernetes, which renders a data center into an infinitely flexible fabric of resources. From operating systems to mobile apps, we’re living in a time in IT history that’s exciting like it’s never been, and it’s because people are sharing what they know.

And yet despite some notable progress at the margins, the US Government has failed to embrace open source as industry has. Ben Balter of GitHub produced an excellent explainer for this, citing all the usual suspects, including FUD, procurement, and culture. In short:

In a world of competing priorities, government employees will likely choose to move on to the next citizen-facing project, rather than spend potentially months combating the bureaucracy’s immune system…

He concludes by pleading for more “enterprise-y” open source companies:

the open source community needs to step up its game, both in terms of what it offers — moving past the perception that open source is written by hobbyists — and the reception government code receives. On the supply side, there’s a tremendous business model in being the suits behind any one of the Internet’s most popular open source projects…

I couldn’t agree more. There is widespread confusion over “open source”, the different ways it can be used, and what it demands from an agency’s IT strategy. “Open source” has been freighted with twenty years of emotionally satisfying but caricatured ideas of what it can and cannot do for a government agency. Depending on who you ask, it’s fetish that can cure government IT of its ills, or a terrifying confusion of unacceptable risk and blown budgets. Government IT is polarized, and when a community is polarized like this, we’ve lost some important nuances.

It’s especially important to distinguish between open source products, which are maintained and supported by a company, and open source projects, which are maintained by a community. They are both valuable, but they each demand different workflows, talent, and policies. If you use an open source project without the necessary support infrastructure, you’re asking for trouble. On the other hand, a vendor-supported open source product can mire your team in old approaches, and can become a boat anchor to your work. This confusion over the difference between the two, and the awful consequences of choosing the wrong one, can sour the bureaucracy on open source forever. So let’s fix that.

…digital services teams should consider using open source, cloud based, and commodity solutions across the technology stack, as these solutions have seen widespread adoption and support by the most successful private-sector consumer and enterprise software technology companies.

– U.S. Digital Services Playbook

Open source projects are operated by communities. They are innovative and agile. Maybe they don’t know what they want to be when they grow up. As the community explores a particular problem, the software changes in shape and purpose. Projects are, in many ways, experiments. Sometimes, those experiments fail – which is a good thing – and when they fail, you’re responsible for fixing it. In many ways, a project makes you the vendor. That responsibility is the cost of free access to good ideas, the freedom to experiment, transparency, and a powerful collective effort.

At a certain point in a project’s maturity, a vendor will arrive and package up a version of the project that’s stable, safe, and predictable. The offering will have steady release cadences, support agreements, and certifications. The project has become a product. Over time, different vendors will make different improvements to their version of the project to win your business, but the project has largely settled down. You know exactly what you need this software to do, and it does it. Products can be less risky for an organization, because the vendor is contractually bound to help you if you have a problem. Competition keeps the cost down. In other words, you pay for open source products to be as boring as possible so you can focus on the really interesting stuff.

The wonderful thing about open source is that it offers this choice, a continuum between more conservative, stable, and risk-averse products, and faster, more agile software projects – and everything in between. This distinction was enshrined 15 years ago in OMB circular A-130:

(b) Acquire off-the-shelf software from commercial sources, unless the cost effectiveness of developing custom software to meet mission needs is clear and has been documented;

Unlike proprietary software, open source serves both needs. A product from an open source vendor is for the commercial “off-the-shelf” problems, and open source projects are appropriate for the “custom software” case.

Once you’ve made this distinction between project and product, you still need a way to choose the best one for your needs. Fortunately, plenty has been written about identifying healthy open source projects. OSS Watch has an excellent discussion of health measurements for open source that I can recommend.

As for products, happily, they can be evaluated just like projects. They just express successful attributes in different ways, and are actually not all that different from evaluating other software. Where a project should have many stars on GitHub or a vibrant mailing list, products should have customer references and a healthy partner ecosystem. Where a project’s quality can be measured by its test coverage or bug turnover, a product’s quality can be measured by certifications, like FIPS 140-2 or Common Criteria. Here’s a sketch of what I mean:

	Projects	Products
Reputation	Many GitHub stars	Respectable customer references
Quality	Thorough automated tests	Many 3rd-party certifications
Reliability	Lots of mailing list activity	SLAs
Vitality	Large, friendly community	Leadership in the upstream project
Viability	Many dependent and adjacent projects	A healthy ecosystem of partners
Freedom	Licensing terms you enjoy.	Commitment to open standards and facilitates an easy exit strategy

As A-130 suggests, pressing a project to do a product’s work can be disastrous. If I force a veteran sysadmin to use an operating system that turns inside-out every week, I’m going to blow my operations budget and their patience. The reverse is also true: if I force a web developer to use a three year-old productized version of Ruby on Rails, they’ll probably quit and work for another shop that understands how quickly these frameworks evolve.

So the pervasive notion that open source is only for hobbyists, or that it’s not “ready for the enterprise”, misses the point. Instead of “open source” and “not open source,” It’s more useful to think about “solved problems” and “unsolved problems.” Open source works great for both: products are for the solved problems, and projects are for the unsolved problems. Because your operation is a living, breathing thing that learns, and gets better at its job, the project-oriented, experimental work should eventually mature and become the work of products. There was a time when people assembled purpose-built Linux distributions and compiled their own web servers. Those days are thankfully behind us, thanks to this maturity model.

Think about what this means for your mission. Should you be spending the same effort to maintain an operating system and your customer-facing services? Of course not. Labor is expensive and complicated and takes sick days, so you want your talent, time, and attention focused on solving the unsolved problems.

This means building an organization and a set of practices that knows how to take solved problems and standardize them, automate them, and get ruthless about keeping them boring. We’ve been able to do this for years with hardware. I remember one of my first jobs was changing the 12″ magtapes on a backup system. Like an animal, literally: you could train a monkey to do that job. Thankfully, the technology matured, got boring, and now we don’t even bother giving it to a monkey: we make a machine do it. That frees me up to automate another task. So it should be with software.

Compare this vision of the world, this maturity model, to the much more narrow, sick world of the proprietary enterprise software shop. You’re not working together with a global community of innovators. You’re not experimenting. You’re definitely not commoditizing technology, because the vendors are fighting tooth and nail to differentiate. Instead of making your organization smarter, you’re waiting patiently for the next incremental improvement from your vendor.

Just visit the websites of software vendors, they tell you everything you need to know: stock photos of suits shaking hands in airports, and high-fiving at conference tables. They’re not interested in maturity models or commoditization, they’re interested in a successful ELA negotiation and that endless hamster wheel of incremental improvements.

This is why open source is so important. Open source projects give you the tools to get smarter and better understand your problems. Open source products make it easy to take what you learned and bake it into your operation. Together, they give you the tools to build an organization that learns.

Step one is an open source policy and program. I recommend starting with the policy the 18F put on GitHub. Then take a long, hard look at your mission and your portfolio. Where the problems are solved, bring in the open source “suits” to help you control costs and risk. Where the problems aren’t solved, bring in teams like 18F or the US Digital Service, who can help you experiment, and create a better way of doing things with open source.

So when you think of open source, don’t think about cheap knock-offs of your enterprise products. Think about billions of dollars of research and development you don’t have to spend. Think about IT as a maturity model, not a tedious assembly of proprietary products that don’t have your best interests at heart. Your ability to be open to that innovation, to work together with other companies and other agencies, will define your success or failure.