PaaS and Three Cruelties of Federal IT

A construction platform.

The MeriTalk “PaaS or Play” study says 70% of Federal agencies are considering PaaS, and 40% expect to be using it in the next three years. That’s great for PaaS, and great for OpenShift, my company’s PaaS… but it’s a little strange. We think of PaaS as something developers use, and very few of these agencies have actual developers on staff. So why the enthusiasm? In order to understand the appeal of PaaS, you have to first understand how bad life is for us in government IT.

#1: Budget and Execution

The Federal government underspends on new IT projects, and we spend badly. Most of our $80 billion IT budget, about 70%, just keeps the lights on: no material improvements, no new services. The money that remains is spent on new projects that go over budget or over schedule 70% of time. That’s an atrocious record.

Ideally, budget or execution problems would force agencies to triage under-performing projects, but agencies don’t have that luxury. Many of those projects exist because Congress has legislated them, or the Executive has mandated them, so agencies have no choice but to continue investing.

Ideally, this hardship inspires innovative new approaches. It doesn’t.

#2: Procurement

The lack of new approaches brings us to the second cruelty of Federal IT: procurement. Any new IT project tilts against decades of accumulated rules and procedures, all bent on eliminating risk. The RFI, RFP, milestone-driven treatment of IT projects basically guarantees that 70% error rate: it’s a process that assumes you know everything when you start, and that you will learn nothing while you’re doing it. It’s a meatgrinder for good intentions. The results of the system are consistent in large projects: FBI Virtual Case File,, etc. The most degenerate case is the DOD, where projects like the F-35 fighter, the DDG-1000 destroyer, and the Air Force ECCS ERP system are bigger, more complicated, and have all succumbed to the inflexible, multi-year acquisition system.

So budgets are unpredictable, execution is poor, and our procurement system reinforces bad behavior. The only thing that would make this worse is to outsource all the work, ensuring that agencies have no institutional competence in the field.

#3: Contracting Culture

So that’s exactly what we do, and it’s the third cruelty. Contracting culture, while providing some advantages, also provides strong incentives to build from scratch and favor labor over automation. Building from scratch means the contractor keeps the intellectual property and creates a strong incumbency for follow-on work. As contractors metastasize with the programs that employ them, the government grows ever-less capable of proper oversight and governance. The number of government programs that have no meaningful plan for the retention and use of the intellectual property they paid for is appalling.

So this is a perfect mess. Almost every step of government IT is broken in some meaningful way, and each broken aspect feeds on the other. We can certainly make improvements at the margins, like UK-style Digital Government Offices or reform procurement at the edges, but I think agencies have everything they need to address these problems, and that’s what brings us to PaaS.

Platform as a Service

When applications are done well, they are just the really application-specific, brackish residue that can’t be so easily abstracted away. All the nice, reusable components sublimate away… where everybody can collaborate to advance the commons. — substack from “how I write modules

In IT as Manufacturing, I told the story of the B-24 Liberator bomber, and how manufacturing-style standardization and automation, rather than craftwork, can have profound effects on how IT projects are run. I’m convinced this approach can help government IT projects.

The premise is pretty simple: we separate the problems we’ve solved from the problems that we haven’t solved. What we’ve solved belongs in the platform. What we haven’t solved is the workload, which rides on top of the platform.

A construction platform.
A $20B idea.

Platforms are composed of standard components, preferably commodities. Their job is to be predictable, identical, and as cheap as possible. This includes servers, operating systems, Java containers, that kind of thing: all the reusable pieces that enable our workload.

Building platforms allows us to get out of our own way. We don’t build cars from scratch, we don’t build toasters from scratch, so there’s no reason we should be paying integrators to build IT systems from scratch. Years ago, it made sense to pay someone to stitch a bunch of low-level components together. We’re smarter now, we understand these problems better, and we’ll understand them even more in the future. We should have an infrastructure that acknowledges that.

The ideal platform is not, however, a black box or a million-dollar server the size of refrigerator. Opaque, vendor-specific systems like these assume that we’ll never learn anything again. Instead, our platforms should be competitive spaces, where we can quickly take advantage of innovations or changes in the market by swapping out one component for another without disrupting the workload above.

That means a well-defined process for integration, and keeping your barriers for entry and exit on a particular component very low. As an IT organization, we want to manage the components of a platform as we would a stock portfolio. We know what we need, and we’re looking for the best performance at the best price.

We standardize these platform components so we can focus on what actually matters: the workload. We have to do this, because workloads are gnarly and time-consuming to create. Workload development is not an industrial process like the platforms. Workloads are creative processes. Our developers need to pull together all the resources they can, and use blood and sweat to render them into a tool that serves the agency’s mission.

This is what makes them so expensive: without a line in the sand, without the bulwark of our platform in place, that creative process will naturally begin to infect our well-ordered commodity components. Before you know it, every applications is a beautiful and unique snowflake that needs its own perfect hardware, its own deviant version of the database, and so on. This is where complexity comes from, and this is where integrators make a great deal of money.

The goal is for a workload, over time, to get better-defined, better-understood, and much less creative. It should become a standard. Once we’ve reached this point, we take the work and fold it into the platform, where it can enrich other, newer workloads. Just like that, you have a properly-managed lifecycle and maturity model for the software we built.

This might sound an awful lot like an “enterprise architecture”. That’s exactly what it is. The difference is that unlike most attempts at an enterprise architecture, the platform we’re describing exists in the real world, not on paper. It takes the form of documentation, software, artifacts, and processes that your developers and contractors can use to build their workloads. That’s where Platform-as-a-Service comes in.

A PaaS is the physical manifestation of your Enterprise Architecture. It’s what musters our approved components and makes them easily available to developers and contractors. The PaaS can host new tools and frameworks as quickly as they’re invented. The PaaS keeps the creative process (and the lock-in opportunities that come with it) isolated from a well-defined, smoothly running infrastructure.

A PaaS limits our exposure to outside contractors. It mitigates procurement challenges, because we’ve driven complexity from our task orders. It reduces costs because our projects are now smaller, limited to just the workload. In fact, we may not have to procure anything at all: the platform’s paid for, and can host that Very Good Idea without ever talking with a contracting officer or a system administrator.

All together, this means cutting literally months out of the project plan, eliminating a wonderland of redundancy, and drastically improving our governance and oversight. The one-third of respondents to the Meritalk study that admitted to using unapproved tools or software wouldn’t have to sneak around. The 50% who believed they were locked into their current IT choices could be set loose. The 41% who need to update their workloads now have that goal in reach.

That’s how you liberate 25% of the Federal IT budget.


p>In case you were wondering, you can try OpenShift for free and stand up your own OpenShift almost painlessly.