The Secret to Twenty-First Century Software Innovation

We at ZapThink are sure that you are as ecstatic as we are that the whole “2.0” cliche has finally died down. Web 2.0, Enterprise 2.0, SOA 2.0, the list went on and on. At first a marketer’s dream, but soon a marketer’s nightmare, as the term became so watered down that it lost whatever meaning it originally had.

However, this loss of meaning is perhaps unfortunate, as there truly is an underlying trend in the greater technology marketplace that does distinguish a new wave of innovation (“2.0”) from the earlier wave of a dozen years ago (“dot.com,” rechristened “1.0” by marketers who love trite consistency). On first glance, this trend centers on collaboration, and it’s quite true that many 2.0 era innovations are collaborative, but that designation misses the point. After all, much of the technology from the 1990s focused on collaboration as well. Remember groupware, knowledge management, and intranets? All collaborative.

Understanding the true root of innovation behind the superficial 2.0 marketing cliche is the secret to today’s software innovation. If you’re an entrepreneur looking for the right idea or hoping to evaluate the idea you have, then this secret is essential to your success. And if you’re in enterprise IT, understanding this secret will enable you to evaluate innovative products coming to market, so that you can separate the ones that will truly provide value from the pack of dot.com also-rans.

A Review of Complex Systems

You can find the secret to true collaborative innovation in Complex Systems Theory. ZapThink has discussed Complex Systems before, in the context of Service-Oriented Architecture and more recently in the context of SOA governance in particular. Essentially, a Complex System is a system that exhibits emergent properties, which are properties of the system as a whole that the components of the system do not exhibit. Complex Systems Theory is particularly fascinating because it describes many natural phenomena, from the human mind to the growth of living creatures to the principle of friction. Furthermore, if you assemble a group of people large enough, they become a Complex System as well, which explains simple emergent properties like a stadium of people doing the wave, to more subtle ones like the wisdom of crowds.

It’s important to realize that Complex Systems are actually systems of systems. The individual elements of a Complex System are themselves systems, which in turn may be Complex Systems in their own right. However, the individual component systems do not exhibit the emergent properties that the larger Complex System will offer.

While a large enough group of people will constitute a Complex System in its own right, for our purposes we’re looking for innovation in Complex Systems that include some software subsystem. Software, however, is not the only subsystem; people must also be a part of the Complex System we’re looking to create. In fact, understanding this basic principle is at the center of the secret to collaborative innovation.

Avoiding the Wrong Approach

Traditional software innovation focuses predictably on traditional systems, as opposed to Complex Systems. To design a traditional system, start with the requirements for that system and build to those requirements. As a result, the best you can expect from a traditional system is that it does what it’s supposed to do.

The emergent properties that Complex Systems exhibit, however, may be unpredictable — at least, before we build the system to see what behavior emerges. The stickiness property of Velcro, for example, is familiar and predictable to us now, but it took a great leap of innovative imagination to look at the little hooks and loops that make up Velcro and see that enough of them put together would give us the stickiness that makes Velcro so useful. The behavior of stock markets, in contrast, is inherently unpredictable, although it does follow certain patterns that give technical analysts something to base their theories on. But if technical analysis really predicted market behavior, there would be a lot more billionaire technical analysts in this world!

The wrong approach, therefore, is to build to a set of fixed requirements, which will tend to eliminate emergent behavior rather than encourage it. This limitation gives startups an advantage, because most traditional information technology (IT) solutions follow traditional systems approaches, which limit their flexibility. For example, traditional integration middleware follows a “connecting things” approach that leads to reduced agility over time, while SOA (properly done, not the fake SOA peddled by the middleware vendors) follows a Complex Systems approach that yields emergent properties like business agility and business empowerment.

So, how do you avoid the wrong approach? Don’t build anything that does what it’s supposed to do. Instead, build something that will lead to surprises. Not every surprise will be useful, to be sure, so you may have to try a few times before you find an emergent property that people will actually appreciate. Also, remember that any system of systems that has component systems that consist solely of technology will most likely be the wrong approach, as the only surprises you’re likely to end up with are bad ones, namely, that it doesn’t even do what it’s supposed to do.

How to Look for an Opportunity

The key to successful collaborative innovation is to realize that humans are part of the system, not just users of the system. While people are unpredictable individually, they are always predictable en masse. Your product development, therefore, must work at two levels. You must think about individual human/technology subsystems as well as the complex human/technology systems that emerge when you have enough of the subsystems in place. Remember, you can influence human behavior at the Complex Systems level by introducing technology at the component system level. To generate emergent properties at the complex systems level, then, you must introduce some element of unpredictability at the component level.

A perfect example of this phenomenon is Twitter. At the component level we have users entering up to 140 characters at a time into a large database that is able to display those tweets based upon various search criteria, the default being your own tweet history. However, if that description were all there was to Twitter, it would never have become the sensation it did. It was the fact that people could follow other people, and that people realized they could use hash tags to designate keywords and “at” people with the “@” symbol that introduced an element of unpredictability into the behavior of the individual user/Twitter subsystems. Scale that up to millions of users, and you have the emergent properties inherent in Twitter trending and other aspects of the Twitterverse that make Twitter such a fascinating tool.

Other examples of successful Complex Systems innovations include:

• SOA governance — human activity-centric governance processes supported by a metadata-driven infrastructure enable SOA implementations to exhibit the business agility emergent property.
• Viral marketing — one person uses a viral marketing tool to tell his or her friends about something cool in a way that encourages them to do the same, leading to the emergence of popularity across large populations.
• Semantics tooling — our computers aren’t smart enough to understand the meaning of data, so to effectively automate semantic integration requires human/software subsystems, which lead to the emergence of automated semantic context.
• Crowdsourcing — Ask a random person to do something or provide information and there’s no telling what you’ll get. But use a crowdsourcing tool to ask enough people, and you’ll get the emergent property of the wisdom of crowds, where the crowd is able to arrive at the correct answer or solution.
• Anything with the network effect — One fax machine or Facebook when it had one user are entirely useless. Two fax machines or Facebook with two users are almost entirely useless. Up the number to three, four, five…still pretty damn useless. But at some point, there’s a critical mass of users that makes the solution valuable, and you get the emergent property that more people want to join all of a sudden, where before they didn’t.

The ZapThink Take

Our core advice to entrepreneurs, therefore, is to come up with a simple tool that is simple when you put it in front of individual users, but yields some kind of unpredictable behavior that when scaled up to large numbers of people delivers an emergent property that people will value. Keep in mind, however, that you hope to be surprised by the result. It may or may not be an emergent property that people will end up wanting, and thus may not be the basis for a viable business model. There is an inherent element of experimental innovation involved, which makes product development inherently risky. On the plus side, however, that risk gives you a barrier to entry, especially from large vendors who sell predictability.

The Complex Systems we’re describing are inherently collaborative, because you’re including people in the system, and the unpredictability you seek results from allowing them to interact in some way. But remember, the converse is not necessarily true: not all collaborative systems are Complex Systems. Any multi-user application in the enterprise, for example, is collaborative on some level, but if the vendor didn’t build the right unpredictibility into it, then it won’t exhibit emergent properties. Those sorts of tools don’t deserve the “2.0” appellation. To create a truly innovative collaboration-based solution in today’s software environment, you must create a Complex System.