The Best SOA Pilots Don’t Get the Services Right

When Thomas Edison was in quest of inventing the electric light bulb, he didn’t get it right the first time. In a well-documented exchange, Edison was told that he had failed 700 times in his attempts to make the breakthrough he sought. In response, he said, “I have not failed 700 times. I have not failed once! I have succeeded in proving that those 700 ways would not work!” What we can learn from the king of perseverance is that failing often and quickly is one of the best ways to increase the likelihood of success, and do so sooner.

However most developers and architects, especially those building Services, want to minimize the number of times they spend rebuilding anything — and for good reason. Development is expensive. Architecture takes time. So, doing things over is rarely an option. Indeed, while we often hear the refrain, “I don’t have time to do it right, but I have time to do it over” we tend to shake our heads in disdain. Yet, perhaps there is a kernel of truth in that refrain. From the Edisonian perspective, trying to get things right increases the penalty for failure. If you can only afford to do things once, it had better be right or you just wasted a lot of time, money, and resources.

On the contrary, one of the fundamental ideas inherent in SOA is the notion that things keep changing. If it’s not the business and their requirements, processes, policies, or metadata that change, it’s the underlying technologies, implementations, and schema that do. The enterprise is constantly shifting. Yet, despite these changes business must go on. In this reality, how can one even hope to get it right the first time when the assumptions that presuppose what’s believed to be correct are guaranteed to be wrong within a short period of time?

Yes, there’s no point to trying to get the Services right, get the infrastructure right, build the right governance, quality, and management systems and processes, and associated metadata, and trying to lock them down so that they can’t change. If you know that they will change in advance, why spend months or sometimes years trying to build the perfect Services when the business forces promise such change as to ruin your plans? Beat the urge of perfection by focusing on failing your way to success. When things keep changing, there’s no way you can always be right. So fail often to succeed sooner. And to do that, you need to reduce the cost of failure so that failures are just successful proofs of how not to do things.

Planning to Get things Wrong
When we tell our advisory clients that it’s not important for them to get things right at the start, we often get significant resistance and push-back. “Are you telling me that I don’t need to build functional Services? Are you saying that I don’t need any infrastructure to run those Services? Didn’t you tell me that I needed to get the Governance/Quality/Management (GQM) infrastructure right to even start with Services?” In response, we often tell them that of course they need to build functional Services on a functional SOA infrastructure using the best practices available to them. From the get go, the Services built need to have both functional and non-functional correctness, otherwise they’re not meeting any business requirements.

The point is that we need to see our attempts to meet those functional and non-functional requirements as simply that — attempts. The implementation might be wrong in its optimal approach to meet the requirements. The business might be wrong in its definition of its requirements or business process. Or they both might be right — for now, before things completely change in a shorter time frame than expected.

And there’s so much that can go wrong from a Service-oriented perspective. The Service contract might not be suitable for a future need. Services might have been designed at the wrong level of granularity. A Service policy that was thought to be immutable now has changed. The SOA infrastructure solution that you spent nine months developing an RFP, evaluating, purchasing, and then implementing, might prove to be completely inappropriate for a new scenario the business desperately needs. You built something you thought would last, and you made it such that redoing things is now impossibly expensive and time consuming. Things have changed, your solution doesn’t work. Who’s fault is it? Yours. Because you didn’t design to fail. You designed to make failure expensive.

Reducing the Cost of Failure
There are four primary ways to reduce the cost of failure, or in other words, increase the overall likelihood of success for any system that is continuously changing:

  • Allow for ambiguity — One of the hardest things for developers and architects to do is to design for unknown requirements. Certainly no one can be asked to do a task that the requester themselves doesn’t know. But we are often asked to do tasks that are ill-defined. Many times a business unit wants to see an implementation so that they can know better what they actually want. That’s what prototypes are all about. A prototype, like a pilot, is a model of the actual deliverable, but produced in such a way that it can be easily changed or disposed of without significant cost.Design all the aspects of your SOA to be prototypes, and you’re allowing for ambiguity. This isn’t just restricted to Service contracts — this also applies to SOA infrastructure. Why bother spending 12 months implementing a $5 million ESB product when you don’t even know how the Services will be used? Start with a prototype SOA infrastructure solution. But planning to change doesn’t mean wasting your time, either! Start with a best guess for the Services you think you need, realizing that the contract, level of granularity, aspects of governance, management, security, and performance will change. Show the business the results so that they can use it to better define their requirements. Prototype faster, cheaper, and better, and you will be successful sooner.
  • Design for flexibility — When you see a part of the technology or business change often enough, you start realizing that you need to accommodate the possibility for that change when you design the system in the first place. From a SOA perspective, we allow for this variability through the power of abstraction. We talked about ways to implement such variability in our Seven Aspects of Loose Coupling ZapFlash and in many subsequent Webinars, white papers, and articles. Keep things abstracted and build for flexibility, and you won’t be disappointed when things change. Start with candidate Services, contracts, metadata, schema, and processes, and know that they will change. Be pleased when they do.
  • Behave iteratively — Design and implementation are not enough to guarantee success in the face of continuous change and ambiguity. You have to also align your organization, methods, practices, and even budget to enable continuous change. We have long advocated the movement away from waterfall styles of development to more iterative styles because iterative styles naturally force organizations to plan for change both in technology as well as process and budget. As we recently discussed, moving away from a project-centric mentality helps quite a bit, as well as building organizational excellence in accelerating the number, quality, and efficiency of each iteration.
  • Measure frequently — Allowing for ambiguity, designing for flexibility, and being iterative aren’t enough to guarantee success — you need to make sure you are heading in the right direction! This is where Champions of Failing Your Way to Success can allay the fears of the Perfectionist Crowd. Every iteration and implementation needs to be measured against success criteria. Those criteria might be functional as well as a non-functional. They might be relevant to business metrics such as profitability or technology metrics such as performance. Only by knowing what to measure can you know what to manage, and in turn, know that you are failing your way to success.

I think we would be misleading you if we told you it would be easy to accomplish all the steps above. There’s so much in the Business-IT environment that conspires against the “fail your way to success” approach to SOA. For one, IT budgets are still rooted in the “get it right or else” style of pay-and-pray implementation. There’s simply no leeway in the budget, schedule, or resources to keep doing things over. Our answer: change your budgeting, time scheduling, and resource allocation, because you’ll keep doing it over anyways. Show me one company that managed to get it right the first time and keep it that way, and you’ll win the business Nobel prize.

The second major force that conspires against a redo-centric SOA approach are SOA consultants that make it impossibly expensive or impractical to iterate. When you ask a consultant to implement your SOA, you are telling them that you know what you want, and you’ll be happy if they provide results to match your expectations. But what if your expectations change? What if the deliverable is ok today, but wrong tomorrow? What if you don’t exactly know what you want in the first place? What if the consultant doesn’t really understand SOA much either, and you are their “guinea pig” (happens more often than not)? Outsourcing, subcontracting, and using third-parties tends to favor the pay-and-pray approach. Smart consultants realize their job is to help facilitate the iterations and change their implementation and business models to match. Stay tuned, we’ll write more on this subject!

The ZapThink Take
The whole point of SOA is to enable agility through the consumption and composition of heterogeneous, abstracted Services. The more that we can enable change without having a high penalty cost for that change, the more we enable business agility. If you agree with this viewpoint that SOA is about enabling continuous change, then it should be obvious that “getting things right” at the start is simply not an appropriate goal for a SOA effort. In fact, the goal might be just the reverse — build Services that you know you won’t be right and then make sure the architecture enables you to change them without breaking anything.

If you find that changing Services breaks things at a wide range of levels of abstraction, then you know there’s a flaw in your architecture. Proving that you can simply build Services isn’t enough to prove you can implement SOA. Indeed, all that you have proven is that you can build Services. To prove that you can implement SOA, you have to prove that things can change without breaking. And to do that, you have to intentionally do things wrong to prove that you got it right.

Download the Full The Best SOA Pilots Don’t Get the Services Right Report Here