The Governance, Quality, Management Trifecta: Making Loose Coupling Work
With Service-Oriented Architecture (SOA) well established in the common parlance of IT, companies are now grappling with the some of the most challenging issues in making SOA work. The companies that are well on the path to SOA adoption know full well that the technical challenges of building and exposing Services are less significant than the hurdles of building loosely coupled, business-relevant Services leveraged across their continuously changing business processes. Indeed, the challenge of making loose coupling a reality is only surpassed by the even-greater challenges associated with organizational and cultural adoption of SOA.
While SOA abstracts the complexity associated with heterogeneous, point-to-point integration and tightly-coupled application logic, it introduces a different kind of complexity: the management of distributed, loosely coupled, and dynamically composable Services. Over the past few years companies developed a number of approaches for dealing with this new form of complexity: management solutions that isolate failure and provide mechanisms for abstracting end-point differences, quality solutions that provide mechanisms for assuring changes can be propagated in environments of significant change, and governance approaches that provide oversight into the development of Service-oriented systems, mitigation of change and version management issues, and enforcement of policies core to the operation of the business as a whole.
ZapThink has written numerous times about each of these critical areas of SOA governance, quality, and management, but only recently have we realized that each of these distinct market segments in their own right are really different solutions to the aspects of same problem: making the illusion of loose coupling a reality. In combination, SOA governance, quality, and management form a perfect trifecta that can make the perceived difficulty of loose coupling in a continuously changing IT and business environment a reality.
The Connection Between Runtime Quality and Management
One part of this trifecta is the connection between runtime SOA quality and SOA management. The challenges of managing high quality, Service-oriented systems is fully explored in the Quality SOA ZapFlash. To summarize, it makes very little difference if one Service works in isolation if some aspect of how its composed, or some change to metadata impacts the performance of the system as a whole. In essence, unit testing an individual Service implementation is wholly inadequate to determining whether the Service actually performs in a composite environment of metadata-controlled Services.
The only way to effectively insure SOA quality is to do so continuously, measuring the quality not only of a discrete, atomic service, but also all related metadata, composition logic, policy, and underlying schema continuously in production. Most firms are not familiar or comfortable with this idea of in-production, continuous testing of their systems. However, as we have explored in numerous ZapFlashes including Grappling with SOA Change and Version Management and SOA Quality and Governance: Satisfying the Metarequirement of Agility, the architecture is the business, and since the business is continuously changing, a Quality Assurance (QA) model that requires duplication of the environment in order to insure quality will be enormously expensive, impossible to manage, and ineffective.
Making runtime quality actually work in production requires a mechanism to isolate failures from having recursive and unpredictable effects. This is done by implementing test modes as matters of Service contract and policy, and minimizing the side-effects of Services in a “test mode” such to avoid any unnecessary commitment of data or action. The ideas of policy-driven test and runtime enforcement of quality overlap those of runtime Service management. Many SOA management solutions provide policy enforcement, exception management, failure recovery, and root-cause analysis. Pairing the capabilities of runtime quality tools that facilitate the process of versioning with solutions that minimize the impact of those changes is a natural fit.
Furthermore, there is a “management-quality feedback loop” that exists between tools and approaches to management that provide visibility when systems are approaching an undesired state and mechanisms that allow incremental testing and quality management of the system as a whole. This feedback helps to guarantee loose coupling by making sure that any Service-related changes don’t break things — a fundamental requirement of loose coupling.
The Connection Between SOA Management and Governance
Likewise, there is a connection between SOA Governance and Management that facilitates loose coupling. SOA governance has three distinct, but related parts: design-time governance that provides rules and policies for creating, exposing, and consuming Services, run-time governance that governs the behavior of Services in production and the performance of the architecture as a whole, and change-time governance that details how organizations can effect changes in the overall system with the least disruption to the existing business and its policies.
SOA Management offerings can address aspects of policy enforcement, rules-based routing and decision making, and exception handling. As such, policies managed by governance tooling can be enforced at runtime through active SOA management systems. While SOA governance tooling such as registries and repositories serve as systems-of-record to manage the metadata associated with Services, SOA management approaches provide a means to make sure that Services consumed in production comply with the policies established in those systems. SOA management tooling can also detect and prevent the occurrence of rogue Services and inspect Service interaction further enabling the value of SOA governance approaches. In addition, runtime SOA management helps with change-time governance issues by enforcing governed and approved changes in the distributed environment while minimizing quality and performance issues.
Furthermore, effective SOA governance requires effective management to provide the visibility required in runtime systems to feedback into the governance process. This “management-governance feedback loop” is part of making sure that not only do companies govern their overall SOA efforts, but also provides effective control and management without overly constricting the agility of the architecture. This feedback loop helps to guarantee loose coupling by making sure that business requirements for change don’t have an adverse impact on the behavior of the whole system, by decoupling the logic of the business from the implementation as it exists at that point.
The Connection Between SOA Governance and Quality
Implicit in the discussion above is the idea of change-time management and governance. To prevent the SOA Butterfly Effect, companies have to manage and govern change effectively so that no change has a chaotic effect on the complex environment of Services. This means not only catching and preventing failure through management approaches at runtime, but also staging and testing those changes in through runtime quality and testing approaches.
Design-time governance also requires that companies enforce Service development practices before uncontrolled Services permeate the IT environment. Effective integration between governance and quality approaches makes that enforcement possible by providing the visibility into deviations from established policy and methods for limiting the spread of Services that are non-compliant.
The challenge of maintaining continuous quality in a continuously changing system is an aspect of maintaining effective governance, and as such, the combination of SOA governance and quality tooling and approaches serve to make that problem more manageable. SOA quality tools provide feedback to governance systems by indicating how policies and changes impact the overall system, and likewise, governance systems and approaches feed into the quality lifecycle by providing continuously changing constraints that impact Services at design and run-time. This “quality-governance feedback loop” furthers the realization of loose coupling by making sure that any design-time change has no impact on running systems and that Service consumers do not need ot hardcode governance rules into their Service consumers or providers.
The Emergence of the SOA GQM Suite
Companies have put too much emphasis on solely Service development, runtime execution environments, and inter-Service messaging as the basis of their SOA infrastructure. In reality, handling the runtime aspects of Service infrastructure is a trivial part of making SOA work. Indeed, the sort of infrastructure most companies are implementing for their Service-oriented efforts can just as readily create tightly-coupled Web Service APIs on top of legacy integration middleware as they can facilitate the building of loosely-coupled, composable Services. Since companies are looking for the latter and not the former as the raison d’etre for SOA, it is imperative that companies place significant effort on the governance-quality-management aspect of SOA infrastructure and not just how to expose a Service interface and run a Service implementation. This is not to say that SOA infrastructure, such as some flavor of Enterprise Service Bus (ESB), is not necessary, but rather that it is not sufficient to guarantee the loose coupling and composability in an environment of continuous change that companies desire of their SOA implementation.
So, companies should focus on the kind of SOA-specific enablement technology that is central to the goals of SOA: the SOA Governance Quality Management (GQM) suite. The suite might not necessarily be implemented as a single-vendor product solution, even thought it seems the market might actually be converging in this manner. Regardless of whether it is implemented as a single-vendor solution or a collection of best-of-breed applications, the SOA GQM suite is the collection of tools and capabilities that facilitate all three feedback loops mentioned in this ZapFlash, and illustrated below:
ZapThink is already starting to see this complete feedback loop become a reality in successful SOA projects. Right now, companies should focus on addressing each part of the GQM suite by making sure that all the feedback loops are implemented. Whether solved through home-grown solutions, best-of-breed vendor solutions, or packaged offerings that aim to provide the whole suite of offerings, companies need to properly address their GQM needs if they ever hope to realize the benefits of SOA.
The ZapThink Take
Companies that are truly interested in the benefits of loosely-coupled, composable, business-oriented Services that achieve a flat-cost of change in an environment of heterogeneity and continuous change would be well-served to focus the technology-specific part of their efforts not solely on the ESB as the lynchpin of their SOA success, but rather on the GQM aspects. Addressing all parts of the GQM feedback loop is beneficial to making the illusion of loose coupling a reality. In addition, what makes the GQM space more compelling is that it represents new technology investment that most companies haven’t already made.
While much of today’s ESB investment is a rehash of existing infrastructure and middleware that is already in place, the same cannot be said of GQM tooling. Investing in GQM in prior architectural and technological evolutions of IT simply was not necessary in an environment of tightly-coupled, proprietary systems, and lack of composable systems. In such environments, testing is a QA activity done only between development and deployment, management something done only once systems are deployed, and then only in a monitoring capacity, and governance an afterthought altogether. However, the compelling value of loose coupling and composition that SOA represents gives companies the opportunity to not only get the architecture right, but also make the right investments in their systems to allow for the sort of governance, quality, and management they probably should have had all this time.