Middleware: Part of the Solution, or Part of the Problem?
I’m sure you’ve all heard the old Henny Youngman joke: “the patient says ‘doctor, it hurts when I do this.’ The doctor replies: ‘Then don’t do that!’” In a similar way, today’s IT systems are a complex assortment of moving parts that cause recurring, chronic operational pain so intractable that many IT organizations now simply put Band-Aids on the problems without solving the underlying condition. Of course, we are talking about the recurring costs and risk that inflexible, tightly-coupled integration approaches cause.
Middleware, of course, is at the center of this discussion of integration pain. In many instances, middleware has helped businesses solve tough IT problems, but in other situations, middleware has simply contributed to the problem, compounding the pain in such a way that simply makes companies want to avoid it in the future. So, how can we understand middleware in the context of Service Orientation? Is it part of the solution, or part of the problem?
Unfortunately, the term “middleware” has come to mean many different things. In the most general of senses, middleware is software that intermediates functionality or data between two otherwise independent and separate systems. In most instances, middleware doesn’t provide the actual application that consumers use or the systems of record that store the data and content information. Rather, middleware is essentially the glue that holds together the disparate systems that provide the user interface, application logic, and data storage that distributed applications need to fulfill business requirements. Glue is good when you need to connect two systems together to work, but apply too much glue, or apply it to the wrong places, and you quickly find yourself stuck!
Middleware: The Good Kind
Companies particularly require middleware when systems have proprietary interfaces and thus require tightly-coupled interactions to extract even the most basic of information from them. The rapid build-out of IT capability in the last two decades led to a proliferation of such isolated islands of business functionality that required companies to unlock the information from the tight grasp of their proprietary APIs in order to provide value to their organizations. Businesses therefore had a hard time getting these systems to do what they wanted, let alone getting them to communicate with other systems to meet new needs.
Middleware, including Enterprise Application Integration (EAI), served a role in helping companies meet the urgent needs of eBusiness, Y2K, and enterprise application development. Such EAI vendors provide data or application access capability usually in the form of proprietary or system-specific messaging platforms and an assortment of pre-built adapters that provide bidirectional connectivity to many types of applications and data sources, such as enterprise software applications, databases, file systems, directories, as well as mainframe and other legacy applications. In addition to basic integration functionality, many vendors provide additional capabilities for performing data transformation, business process modeling and management, message transformation and routing, and a variety of business logic and rules engines. However, this functionality doesn’t help companies solve their chronic IT problems, as we will soon see below.
Middleware for your Middleware
One of the biggest problems of tightly-coupled integration is that number of integration or interconnection pathways that must be established grows geometrically (or n-squared) with the number of systems to be integrated. While some EAI and bus-type integration approaches solve the geometric part of the scaling problem, they don’t address the tightly-coupled nature of the system relationships. This means that when things must change, the cost and complexity of making that change is still significant. In addition, the integration problem is compounded by the fact that most companies perform integration in an ad hoc manner, narrowly focusing on short-term integration needs rather than long-term solution effectiveness. The end result is a tangled web of tightly coupled, brittle integrations that are increasingly expensive to maintain and update.
The bigger problem, however, is that companies now think of distributed computing entirely in terms of connecting systems. Rather than focusing on the creation of business logic that meets business needs, companies now create layers of middleware. Companies now have to deal with the problem of integrating their disparate middleware implementations throughout their enterprise — middleware for their middleware, if you will. They must now spend millions more to manage and integrate the middleware that they have glued in place so well that it is virtually impossible to change or remove.
It simply makes no sense for companies to spend money to get their middleware to work well with other middleware. Why should companies have layers upon layers of middleware, each dealing with a specific tightly-coupled integration problem? Using middleware to solve acute, localized integration issues is one thing, using it to solve chronic problems that relate to business agility and ongoing flexibility is another! Companies don’t need more or better middleware to make their systems more agile. Instead, they need to step back and think through how to build, run, and manage distributed computing infrastructures that are inherently flexible and agile. What they need is architecture, in particular, Service-Oriented Architecture (SOA).
Moving away from an Integration-centric Mentality
Part of the challenge of moving to SOA is that it requires a shift in the way that organizations think about their IT resources and how they develop and integrate business functionality. It is far simpler to implement, install, and deploy a system in isolation and think about how to connect it to other systems in the enterprise after the fact, than it is to come up with a proper architectural approach ahead of time that factors in heterogeneity and reusability. However, the movement to SOA requires this sea change in thought. It will be a difficult task indeed to build loosely coupled Services if you are thinking with an integration-centric mentality. While middleware is still a necessity for most companies, especially when it comes to dealing with the connection between proprietary APIs and Service interfaces, there is no reason that companies should add middleware to their middleware.
Instead, SOA replaces the integration-centric mentality with a Service-centric mentality. In this view of IT, the Services form the crux of the infrastructure. The business can then compose Services into processes and thereby manage business logic in a declarative fashion, while the IT department is responsible for the non-functional attributes of the technology underlying those Services (making sure they’re available, scalable, and otherwise meet the service-level agreements and other policies that apply to them). The act of integration in this world-view moves to the process level, where business analysts and business process architects connect Services to one another as they compose processes. The technical infrastructure that enables this composition to take place therefore no longer centers on integration in the sense of connecting systems or applications together, but rather on building and supporting the Services that the business needs.
The ZapThink Take
Fundamentally, in an effective SOA implementation, integration is a byproduct, rather than an enabler of the architecture. This difference distinguishes SOA from other distributed computing architectures that came before it, like client/server and n-tier. Both of those earlier architectures specified logical tiers and the connections between them, requiring middleware to be the glue that created those connections — and the more tiers, the more glue. SOA reduces the quantity of glue needed dramatically and basically replaces it with the architecture itself. Architecture therefore plays a much more active role than before, moving from being solely a design time activity into the runtime, providing the discipline needed to move integration, as well as business logic, into the hands of the business.