Failing Faster: Lessons from the Open Source Community

[I first presented this at the GTC Southwest conference in Austin on February 13, 2010.]

Hundreds of thousands of open source software projects are capturing an unprecedented amount of innovation and effort from contributors around the world. These projects vary in quality, but represent, in the aggregate, one the largest collective efforts in the history of the world. Even more remarkable: the result of this collective effort is freely available to anyone with an Internet connection.

There are dozens of articles, papers and blog posts that can tell you how to best consume open source software. The best practices for using open source software in an organization are fairly well known, so I’ll only spend a portion of my time on that. The more interesting question is how an organization can incorporate the principles of open source development to function more effectively. That is to say: how can I get other people to help me solve my problems?

Who are these people, anyway?

"Anti-War Anarchist Girl" by knobil, licensed CC-BY 2.0

To answer that question, we need to first understand open source developers. Who are these people? Some are under the impression that these are idealistic freedom fighters, slaving away on a computer in their mother’s basement for the good of mankind. Others believe open source developers are anarchic socialists who believe that property is theft and have an almost religious conviction that all software should be free. Still others believe open source developers are anarchists, bent on undermining evil corporations with their subversive goods. The truth, of course, is more interesting than these stereotypes.

Open source developers work on projects because they want to solve problems. They have an itch they need to scratch.  These are people who work together not because they have an ideological axe to grind, but because they all have a problem, and they all want it fixed. Ideally, the developers themselves are the ones with the problem — but that’s not always the case. More on that later.

“Democracy is the enemy of useful work”

So how does this corps of volunteers work together? Instinct tells us that democracy is the best organizational tool here; we have many people, all of whom have different opinions about how something should be done. If they want to get all the best ideas, they take a vote and proceed by the majority’s opinion.

This is the opposite of what happens. There is no better way to stall progress than voting. If you operate a project by plebiscite, you distract your contributors from the problem, and have them instead focusing on the internal politics of the project. Clay Johnson of Sunlight Labs famously declared, at the start of a CrisisCamp developer event for Haiti, “Democracy is the enemy of useful work.” He exaggerated for comic effect, but his point is sound: the more time you spend deciding, the less time you spend solving your problem.

The Internet Engineering Task Force (IETF), which has a very close historical relationship with the open source community, has a kinder version of this principle: decisions are made by “rough consensus and running code”:

Working groups make decisions through a “rough consensus” process. IETF consensus does not require that all participants agree although this is, of course, preferred. In general, the dominant view of the working group shall prevail. (However, it must be noted that “dominance” is not to be determined on the basis of volume or persistence, but rather a more general sense of agreement). Consensus can be determined by a show of hands, humming, or any other means on which the WG agrees (by rough consensus, of course). Note that 51% of the working group does not qualify as “rough consensus” and 99% is better than rough. It is up to the Chair to determine if rough consensus has been reached. (IETF Working Group Guidelines and Procedures)

This loose process is possible because the participants are not discussing abstractions. The group has assembled to solve a problem, and so focuses on actual, practical solutions as opposed to notional solutions that may or may not exist in the future. Without the distraction of hypotheticals, it is much easier to arrive at rough consensus. Because “consensus” is so crudely defined, the role of the Chair is therefore very important — they decide when consensus has been reached. This is deliberate. With one person making the final decision, the problems of collective action disappear. Back in the open source community, the “Chair” is the leader of a particular project. The project leader has the final word in design decisions, which patches to accept or reject, and so on.

A weapon against tyrants.

It’s easy to see where this process may be abused. A rogue project leader may make self-serving decisions or decisions that alienate contributors. This happens often. Fortunately, other mechanisms balance the project leader’s power. Since the source code for the project is freely available, it is always possible for a dissenting group to “take their toys and go home,” and begin a competing project based on the original. This is referred to as a “fork” of the code. It’s the nuclear option for dissent, since a fork means creating two redundant projects, neither of which benefit completely from the creative work of the developers interested in solving the problem. If you are dissatisfied with the direction of a project, you have to balance the utility of a “whole” collection of developers against your ability to tolerate compromise. In this way, the open source community’s rough consensus keeps contributing developers and project leaders focused on serving as large a community as possible.

So now that we have a sense of who these developers are, and how they work together, we can focus on what makes some of these projects succeed and others fail.

The Feedback Loop

One of the hallmarks of the open source process is how close it brings those who write software and those who use the software. Often, they are one in the same. This very tight feedback loop ensures that developers know almost immediately if they’re headed in the wrong direction. This is in stark contrast with proprietary development models, where software developers emerge from a smoky room, having survived on marketing surveys, focus groups, and their own cunning with a product that may or may not solve the problem. Constant feedback from the beginning of the process ensures that a project stays responsive to its users.

There is no better example of the “smoky room” development model than defense contracting. Because of a complex acquisition process that places thousands of rules, policies and procedures between the user of a piece of software and its creator, developers for the military have a difficult time understanding how soldiers in the field use their technology. That’s why initiatives like General Dynamics’ Edge Warrior are so interesting. Edge Warrior is an attempt to bring the development process closer to the users, and ensure their direct feedback much earlier in the development process. In their words:

Often times, the end products of these investments fall short of user and customer expectations due to:

  • Not understanding the end user needs
  • Not having access to requirements or feedback from users prior to, or during the research and development
  • Not having access or awareness into the military’s top-requested and desired capabilities
  • Lack of awareness into what investments are being made by the DoD in their research labs
  • Lack of access to the actual users and customers who have expressed the need for a specific capability

Participation when you’re not a programmer.

How Fedora encourages participation

Bringing the consumer closer to the producer makes for a better product. When the development process is open, you’re able to take in feedback earlier and more frequently than would be possible in a closed development model. Many make the mistake of thinking that they can’t participate in an open source project because they aren’t developers. This is a pet peeve of mine, because it just isn’t true. There are all kinds of ways to help a project without knowing how to program a computer. The project has a responsibility to make this as easy as possible.

The Fedora project, which is a free and open Linux distribution, has a tool called abrt which makes it dead easy to provide useful feedback. If a program crashes, a little window pops up, asks you what you were doing while the program crashed, and then submits a bug report. As developers work to fix the bug, you’re updated by email. This is a good feedback mechanism: contributors can easily report problems, and get updated as things get fixed.

"Have a problem. Get a bunch of people interested in the problem. When they show up, have something useful for them to do." — Greg DeKoenigsberg

abrt is a pretty sophisticated and powerful tool, and it’s no surprise that it’s focused on bug reports: that’s how developers understand the world. But what if you have an improvement? What if someone wants to contribute to a project in another way? There’s documentation to write, there are translations to be made, and there’s artwork to be created. Your project should make contributions simple for these folks as well. The Fedora project makes this as simple as possible by showing visitors a list of things to do by role.

So whether you’re a people person, a developer, an artist, or a translator, you can easily find something to do. Greg DeKoenigsberg, one of the elders of the Fedora project, often says that the secret to a successful open source project is to “have a problem, get a bunch of people interested in the problem, and when they show up, have something useful for them to do.” It’s a simple recipe, but many failed open source projects forget at least one of these steps.

This works for non-software projects, too, of course. Collaboration can be a powerful tool, but it’s often difficult to decide how best to attract contributors. Recently, the Obama Administration asked Federal agencies to solicit feedback directly from the public. The goal is to start a collaboration with citizens, so that agencies can better serve them by creating a feedback loop very much like those found in the open source community. Many of these agencies have started a website that acts as a kind of collaborative suggestion box. Visitors can suggest an improvement to the agency, and other visitors can see those suggestions and vote them up or down. You can see an overview of all this feedback at OpenGovTracker. This should remind you very much of a bug reporting tool in the software world.

OpenGovTracker on 17 February 2010

Too often, there are real or perceived roadblocks to participation in an organization. This is especially true when you’re trying to cultivate a community from outside your organization. This goes double when you work for the government. There are all kinds of rules around who’s allowed to discuss what with other people, and often government employees are afraid of working with the public. At the Transparency Camp conference in 2009, a representative of the FEC held a session in which the attendees were asked how the FEC could improve the website. The response was interesting. First, there was broad consensus that the transparency advocates in the audience didn’t much care what the website looked like: they wanted the data, raw and unpackaged. Second, an attendee turned the FEC’s question around: how can we help you? What tools can we, the open source community, build that will help you perform your job better?

Getting permission.

This is the dream of citizen participation made real: rather than relying on the government to do the job right, citizens were willing to devote their free time to developing tools that would make their country run better. That excitement quickly turned once the FEC representative responded: he apologized, and then stopped to conversation cold. He was afraid of running afoul the Federal Advisory Committee Act, which prohibits the Federal government from receiving volunteer help. It turns out, his fears were largely unfounded. Because of this uncertainty, though, dozens of contributors lost an opportunity to work together. So it’s very important, especially in government, that the policies and procedures for collaboration are clear and unambiguous. If you hope to capture all the innovation outside your department, your agency, and in the outside world, everyone involved should know the ground rules. Otherwise, collaboration will be stopped cold by fear and misunderstanding.

Building for evolution

The Springfield 1861 Rifle: open source for gunsmiths.

On a technical note, the story of the hopeful FEC contributors illustrates another hallmark of the open source community: modularity. These developers didn’t want to build a website. They wanted to build and share tools with the FEC. In part because of the economics of the collaborative process, and in part because of the way software works, developers tend to avoid writing large, monolithic chunks of software. Instead, they write smaller, more manageable pieces and tie them together. This makes it easier to collaborate, since it’s easier for everyone to understand the smaller constituent problems. This also makes it easier to fail — which is a good thing. The principle is interchangeable parts, which every student of history knows was one of the most important innovations of the Industrial Revolution. One of my favorite examples of this is the 1861 Springfield Rifle:

The Springfield Rifle cost $20 each at the Springfield Armory where they were officially made. Overwhelmed by the demand, the armory opened its weapons patterns up to twenty private contractors. The most notable producer of contract Model 1861 Springfields was Colt, who made several minor design changes in their version, the “Colt Special” rifled musket. These changes included redesigned barrel bands, a new hammer, and a redesigned bolster. Several of these changes were eventually adopted by the Ordnance Department and incorporated into the model 1863 rifled musket.” — “Springfield Model 1861,”

Colt’s improvements to the rifle were made simple by the fact that the rifle was composed of interchangeable parts. This allowed them to focus on specific, iterative improvements to the rifle’s design. If one component isn’t working properly, it can be replaced with another with a minimum of friction.

Mozilla designs for happy accidents.

Modularity also lends itself to new kinds of innovation. Many popular software projects support “plug-ins,” which is software that adds new features or alters the behavior of its platform. Firefox is a familiar example. There are plug-ins that block ads, organize research, synchronize bookmarks — all features that the core Firefox development team would not have the time or ingenuity to develop themselves. Instead, they provided a platform that other developers could use to extend their product. Nobody has a monopoly on good ideas, and the easier you make it for other people to contribute, the more successful your project will be.

The Open Source Way, built the open source way.

You can also see this principle at work in the Open Source Way, a book published by Karsten Wade and Red Hat’s Community Architecture Team. This book describes many of the principles I’ve discussed here, but it much greater detail. The team realized, though, that they don’t have a monopoly on good ideas. So they posted the document as a wiki, which anyone can contribute to — treating the book as a platform for better ideas, rather than an inflexible monolith of received wisdom.

Rather than trying to solve every problem at once, open source projects start small, build on their successes, and very quickly identify dead-ends. They ensure that they’re building not just for themselves, but for anyone who wants to contribute — going so far as to allow other developers to connect with their software without any permission at all. This ensures that the project will stay flexible and responsive to its users.

The importance of rhythm

Cadence is important.

Because open source software works in small pieces, and builds on its successes, it’s critically important to keep a rhythm to the project. If someone visits your project web site, and sees no traffic on the mailing list, or no code being checked in, they’ll assume the project is moribund and avoid using it — even if the software is perfect. This is why obvious activity is so important: it attracts users, and it attracts developers. Good project leaders will ensure that there’s a steady cadence of activity, and that the activity is conspicuous to everyone.

Whoops. This is science.

The open source community came to these best practices through trial and error. So imagine our surprise when we discovered that we weren’t so special, and there was in fact a vast field of academic literature on the subject of collaboration and building communities around useful work. The most cited is probably Cultivating Communities of Practice, by Etienne Wenger, Richard McDermott, and William M. Snyder (Harvard Business Press, 2002). They identify some key components for making communities function properly:

  • Design for evolution.
  • Open a dialogue between inside and outside perspectives.
  • Invite different levels of participation.
  • Develop both public and private community spaces.
  • Focus on value.
  • Combine familiarity and excitement.
  • Create a rhythm for the community.

Sounds pretty familiar, right? Let’s rephrase that in some more concrete terms:

  • Contributors are customers. They’re not doing it out of the goodness of the heart. They want something.
  • When they show up, have something for them to do.
  • A transparent process, run by a despot, works better than a democracy.
  • Make sure everyone knows the rules for collaboration. Especially in government.
  • Let everyone make mistakes. If they don’t, they’ll never innovate. Just make sure they fail small and quickly.


Fail Faster, GTC Southwest, February 2010 (pdf)