Grappling with SOA Change and Version Management

ZapThink frequently speaks of change, since change is the one consistent reality underlying the entire business world. Change, after all, focuses businesses on realizing the benefits of Service-Oriented Architecture (SOA), because the primary reason for embarking on a SOA project is to improve business agility while reducing IT costs in the face of today’s brittle, inflexible, and tightly coupled systems. However, let’s not let the perfume of the SOA rose intoxicate us too much, since implementing SOA itself introduces change every bit as significant as the forces of change that SOA is meant to address. SOA implementations must be built to change, or companies will never be able to realize the agility and cost savings benefits that SOA promises. In particular, a number of change and versioning issues arise when dealing with the sort of living SOA that companies so desire today.

Versioning Service Contract Metadata
The core question that this ZapFlash addresses, therefore, is how to build a SOA implementation for change without succumbing to pitfalls of versioning and change management. Naturally, any discussion of change in the context of SOA must begin with loose coupling. As one of the main benefits of SOA, loose coupling provides developers the ability to freely change the implementation of a Service without having to change the consumers of that Service, and vice-versa. This loose coupling gives enterprises the benefit of flexibility in the face of IT heterogeneity and change. However, loose coupling doesn’t mean that any changes to a Service won’t impact other parties. Indeed, loose coupling only applies if the underlying implementations still comply with and respect the existing Service contract. Changes to the Service contract itself, therefore, can lead to real trouble.

As we detail in our ZapFlash entitled What Belongs in a Service Contract, a Service contract contains both functional metadata that describe how to interact with a Service as well as non-functional metadata that define what conditions and restrictions apply to consumers looking to access Service functionality. As we detailed in that ZapFlash,WSDL is only one small part of the metadata necessary for adequately defining a Service; we also need information on security, process, quality of service, commercial requirements, and other information. In an SOA, the contract is as important as the code that implements the Service. Therefore, when a developer decides to make a small change to any contract metadata, such as acceptable security requirements or quality of service, things can go dramatically wrong. Any Service consumer that makes an assumption about the contract metadata that’s now been changed now breaks, and the entire system no longer functions as expected.

How then, can loose coupling still be a reality? After all, the reality is that just as Service implementations change, so too does the need to change Service contract metadata. So, how can enterprises implement Services in such a way that small changes to Service metadata don’t throw a wrench into the works? By making sure that older versions of Service contracts remain valid until an organization can make sure through proper governance that no consumers will ever use those contract metadata again.

Policy Versioning
Security and policy metadata are part of the discussion on Service contract versioning, but changing the Service contract is only a small part of the policy versioning problem. A policy is a set of rules that apply to any number of Services. Policies typically govern under what conditions consumers are entitled to access Service functionality. Some policies may be security-related, while others are time or process-dependent. Policies can apply to zero or more Services, and it’s important to maintain policies separately from the Service contracts themselves. In many large organizations, the group of people that manages Service policies is separate from the group that manages the Service implementations or the Service contract metadata. As such, policies are likely to change even when the Service contract metadata stay the same.

Independently changing Service contracts and policies dramatically increases the complexity of the SOA implementation. At any moment, an individual can change a policy impacting an unknown number of Services in ways that can instantly cause them to stop functioning. There are only two ways to potentially prevent unintended consequences from changes to policies: by implementing metapolicies that control who can change policies and when, and by establishing adequate governance to control the creation, communication, and enforcement of such policies. In this manner, policies can only be changed by authorized individuals, and even then, only when there’s adequate understanding of the consequences.

However, implementing proper governance by itself doesn’t solve this metadata versioning nightmare. One potential for chaos is the fact that a large organization might implement multiple governance frameworks to allow independent divisions or subsidiaries to control their own SOA implementations, only to later require sharing of such Services across their organizational boundaries. In these cases, even if proper governance exists to limit the impact of changes to Service contracts and policies, the metapolicies that determine how companies create and establish the governance frameworks themselves might change in ways that have unpredictable consequences. Implementing additional layers of governance might in turn require changes to countless numbers of policies and Service contracts, leading to increased cost and complexity. As a result, the proper establishment of an enterprisewide governance framework early in a SOA project is critical for the success of the initiative.

Schema Versioning
An even more basic challenge than dealing with contract and policy change is dealing with changes to the underlying data that Service consumers and providers exchange. What if the Service contract metadata and policies remain the same, but the data schemas change? In this situation, we are once again left with the possibility that Services can’t interoperate or communicate. After all, Service contracts are merely interfaces between systems. The lifeblood that flows through these interfaces are actual data that systems care about. In our ZapFlash Semantic Integration: Loosely Coupling the Meaning of Data, we discussed the fact that we can only achieve an acceptable level of loose coupling if both Service consumers and providers depend on a mutual understanding of the data they communicate. As such, most of today’s Services will depend upon some vocabulary or ontology that all parties engaging in Service-enabled dialogues must share. In the Service-oriented context, such vocabularies appear in schemas that describes the data, which are constantly changing.

In order to address the issue of continuously changing data schema preventing Service interoperability, the management of schema must happen concurrently with the management of all Service metadata. Indeed, we should consider data schema and semantics to be a critical part of the metadata that is necessary for Service interaction. Anytime a schema is changed, all relevant Services should be checked to make sure they can still comply with their published Service contracts. Even governance needs to be extended to data schema management since the reliability of the overall architecture depends on reliable management of underlying data schema.

The ZapThink Take
With all these various ways metadata may change in a SOA implementation , how can an organization audit and log its Service interactions? To understand the challenge of auditability, let’s envision a scenario where an organization stores all its Service messages for later retrieval, possibly for regulatory compliance reasons. Let’s say then that a year later they call one of those transactions into question, which then must be retrieved for verification purposes. However, in the intervening year, the schema, the Service contract, and security policies that govern access to the data have all changed. As a result, the stored message is no longer valid according to the schema that now exists, and furthermore, current users may no longer be entitled to access that message. Even if they could access the message, there’s no way to trace its interaction through the various systems, since the contracts and processes have changed over the previous year.

Companies might consider dealing with this problem by storing all versions of metadata over time, enabling users to recreate the system as it existed at the particular point in time in question. However, this approach is is likely to be impractical. There is simply too much to store and too many changes to correlate to properly trace the history of the message in question and recreate the “reality” as it existed on that date. This lack of auditability is a challenge that current SOA best practices have yet to address.

Yet, these aren’t challenges unique to SOA. Rather, SOA provides a good excuse for organizations to once and for all tackle their challenges of dealing with change management. Today, companies have poor visibility into the challenges of change management because systems are so poorly interconnected and tightly coupled that changes are isolated to specific systems. IT organizations are spending so much effort getting systems merely to communicate that worrying about governance, policies, and versioning is at the bottom of their list. SOA provides a unifying approach for dealing with change not on a system-by-system basis, but rather through a systematic approach that provides a way of dealing with all metadata change no matter the underlying system, process, or consuming application.

Clearly, companies must invest in metadata management if they hope to realize the vision of loose coupling. However, the real issue is that companies need to understand that change is inevitable, even in the world of SOA, and therefore, they must adequately plan to handle such change lest it ruin the primary reason they adopted SOA in the first place.

Download the Full Grappling with SOA Change and Version Management Report Here