Software’s Dirty Little Secret

Since the days of Bletchley Park‘s Colossus, the world’s first programmable computer, the creation of software has been separate from the production of the machines that run that software. For the first time, a machine was programmable instead of purpose-built. In other words, people did not determine the specific requirements for computers before they built them. Instead, programmers wrote software later to meet those requirements. No longer did hardware have purpose-built requirements; instead, people had the meta-requirement of programmability — the requirement that they be flexible enough to meet as-yet undefined requirements at some point in the future.

Today, we use the term high technology to refer in part to programmable machines — whether they be computers or any number of other devices with the computer chips that enable them to meet the meta-requirement of programmability. While computers are high tech, the surprising fact is that software itself is almost always low tech, because software is generally built for specific purposes. Programmers rarely, if ever, build software applications for the sort of general purpose uses that a computer is suitable for. We have done an extraordinary job of building computer chips that can run everything from weather forecasters to video games — but the software itself is only good for one kind of activity or the other.

Software has been low-tech for so long that we’re all used to it that way. After all, software production is still a craft-based enterprise, with experienced practitioners turning out custom software in small teams. Just as when cobblers produced shoes and seamstresses fashioned clothing, today’s software is either entirely custom or appallingly ill-fitting. Custom software is enormously expensive, often requiring teams of developers, and only meets the particular needs of the moment. Meanwhile, commercial off-the-shelf (COTS) software is rarely flexible enough to meet all the needs of its users, requiring users to work within the constraints of the software, rather than the other way around. In any case, regardless of whether it’s custom or COTS, today’s software is of appallingly low quality — and that’s software’s dirty little secret.

The challenge facing us today is whether new, Service-oriented (SO) approaches to software will help to resolve this quality issue. Clearly, there is no guarantee that implementing a Service-Oriented Architecture (SOA) will improve quality — people can still implement SOAs poorly, after all. The real questions are first, whether SOA done right will raise the level of software quality overall, and second, what will it take to do SOA right?

Revisiting Software Quality
Bring up the word “quality” in a business context, and people think of one of two things — either lowering the defect rate as approaches like Six Sigma and ISO 9001 advocate, or formalizing the manufacturing and development processes, as Total Quality Management espouses. Both approaches can apply to the production of software, as well as a wide range of manufactured goods. Developers and testers are the most familiar with the bug hunt and fix approach to software quality, and engineers and IT management within the enterprise often look to the Capability Maturity Model as the established approach to formalizing the processes of creating and managing software.

Unfortunately, both perspectives on software quality fall within the low-tech, craft approach to software production we follow today. Neither perspective addresses the more important and fundamental definition of software quality, which is how well software meets the requirements set out for it. The unfortunate fact is that when people build software to meet specific, pre-defined requirements, it almost always falls short of meeting the true requirements for that software, primarily because the actual requirements for software are typically in a state of constant flux. In other words, our true requirements for software is for it to be high tech — that is, for software to be flexible enough to meet as yet undefined requirements at some point in the future, just as high-tech hardware does today.

Here is the critical point: virtually all software created up to this point is fundamentally of low quality, since it’s all built to meet pre-defined requirements, and those pre-defined requirements are rarely the true requirements for the software. The only way software quality will ever improve is for software to become high tech. The key insight here is that today’s common understanding of what software quality means is lacking, because those definitions apply to purpose-built software. We must take a step back, so that we can judge software quality based upon its flexibility and agility, rather than how few defects it has, or how well-documented the process of creating it might be.

Will Service Orientation Lead to Higher Software Quality?
Business agility is the most fundamental benefit of SOA, where business agility means being able to meet as-yet undefined, changing business requirements, and leverage such change for competitive advantage. Business agility is thus the linchpin of high-tech software. Businesses require software that responds quickly and efficiently to a changing business environment, and SOAs promise this level of agility.

However, it is crucial that we temper this long-term vision with a short-term reality check. It will take many years to achieve the high-tech vision of agile software, and there will be many roadblocks along the way. It won’t be easy, by any means; such a transition will require substantial change, both in technology and human behavior. Nevertheless, today’s SO approaches can provide a good measure of business agility.

What is it about SO software approaches, then, that provide such business agility? What technology advances are we making today that are precursors of the vision of high-tech software? The vision of high-tech software paints a long-term goal, and the following achievements illustrate the progress we’re already making:

  • Metadata-driven functionality. An increasing number of products are moving business logic from the code into the metadata, as metadata management becomes increasingly important to building and running agile SOAs.
  • Declarative “programming.” Instead of true programming, business users are increasingly able to determine the functionality of the software they use through configuration, rather than compilation. Such declarative approaches to determining software functionality move the power over business logic into the hands of business users.
  • The maturation of the discipline of software architecture. Today, architects learn most of their craft on the job. As the discipline of architecture matures, what it means to be an architect will stabilize, enabling companies to plan, design, and produce high-tech software in an industry standard manner. Enterprise architects will increasingly be able to architect their enterprises with business agility in mind.
  • The rise of standards. We are in the early chapters of what will likely be the long story of IT standards, but we’re on our way nevertheless. Standards are one important element to moving beyond craft production of software. Just as standardization in manufacturing led to the industrialization of production, standardization in software will lead to software that is built to be agile.

  • The Service Orientation abstraction layer. Software grows by the addition of abstraction layers that simplify complexity. SO abstracts application functionality, giving businesses unprecedented control and flexibility over the software they use. Such an abstraction layer is thus a necessary first step to building high-tech software.

The ZapThink Take
ZapThink believes that Service Orientation is one of the key mechanisms for leaving the world of low-tech, low-quality software behind, and achieving the goals of high-tech software. Clearly, much work remains to be done, and SO alone does not guarantee quality. However, the trend toward high-tech software is apparent in the progress vendors and IT end-users are making today. ZapThink predicts that we will see a gradual decline in custom programming in the enterprise, as users become able to configure business applications to meet changing requirements. At the same time, companies will need fewer IT developers, as the software they purchase becomes more agile. After all, software vendors should be the companies that produce the software, while enterprises in other lines of business should concentrate on what they do best instead.

Finally, ZapThink predicts the end of craft software, as standards mature and vendors learn how to build high-tech software. Software production will achieve benefits analogous to those that mass production achieved — consistent quality at substantially lower cost, leading to a new era of software quality. As the industry matures and becomes high-tech, software will no longer be judged on how well it can meet today’s requirements, but rather how flexibly it can meet tomorrow’s challenges. Will Service Orientation solve the software industry’s “dirty little secret” of appalling quality? Only time will tell, of course. There’s no question, however, that if vendors become able to deliver software that can truly meet as-yet undefined business requirements, then enterprises will demand such software.

Download the Full Software’s Dirty Little Secret Report Here