Agile Enterprise Architecture is not an Oxymoron!

Of all the feedback we received from our recent TOGAF and SOA guest ZapFlash, perhaps the most intriguing was a comment that TOGAF and the Agile Manifesto were at odds. It’s true that ZapThink has long included Agile best practices in our discussion of SOA, meaning “Agile” with a capital “A,” referring to the Agile Manifesto, so perhaps TOGAF and SOA weren’t as compatible as we were suggesting? We are less versed in the details of TOGAF, however, so it made sense to take a closer look at Enterprise Architecture frameworks like TOGAF to see first, whether they are at odds with Agile approaches, and second, what is the right combination of Agile, TOGAF and SOA best practices for organizations to follow.

What we found is that there is a lesson here for Enterprise Architecture (EA) as a whole. As I’ll be discussing in my upcoming presentation at the Global Integration Summit, many Enterprise Architects see a disconnect between EA and SOA, just as they do between EA and Agile. Such architects disagree that SOA is a style of EA, because it threatens the “traditional” practice of EA, which centers on frameworks and reference architectures and other forms of documentation. Fundamentally, Agile methodologies also threaten this insular world of the Enterprise Architect. The problem, however, is not with the practice of EA, but rather with the perspective of such architects.

Dispelling the Agile Straw Man

Most people are perhaps best acquainted with the Agile approach through specific Agile software development methodologies, of which Extreme Programming (XP) and Scrum are the best known. The attention XP in particular has attracted has led to a broad understanding of Agile that is inaccurate in many ways. Because XP focuses on skunk-works type approaches to software development projects, the common misconception is that XP (and hence Agile in general) means a “cowboy style” approach to programming that eschews documentation and other formalities in favor of furious coding.

Agile really isn’t about cowboy coding, however — it’s much more about maintaining a business focus in the face of change. Let’s take a closer look at the Agile Manifesto, which consists of four core principles:

• Individuals and interactions over processes and tools — Agile emphasizes the role people play in the technology organization over the the tools that people use.
• Working software over comprehensive documentation — The focus of an Agile project is to deliver something that actually works, that is, meets the business requirements. Documentation and other artifacts are simply a means to this end.
• Customer collaboration over contract negotiation — Customers and other business stakeholders are on the same team, rather than adversaries. 
• Responding to change over following a plan — having pre-defined plans can be useful, but if the requirements or some other aspect of the environment changes, then it’s more important to respond to that change than stick obstinately to the plan.

When people think of Agile projects, however, the skunk-works cowboy coding straw man rears its ugly head, which places the Agile approach in a different context from EA generally. Now, it’s clearly true that software development is only a piece of the EA puzzle. When organizations truly follow the Agile methodology in their Agile software development projects, however, they place a focus on the business stakeholders and their requirements, even in the face of change. From this perspective, there’s no reason not to apply such general principles to enterprise architecture as well.

Agile SOA

As ZapThink discusses in our Licensed ZapThink Architect course, there are some Agile principles that are appropriate at the EA/SOA level, but not all of them, and not without some reinterpretation. For example, the second Agile principle above is less a “minimum of documentation” and more that documentation is a means to an end, the end being working software. If we place this general principle in the SOA context, we would say a focus on working, metadata-driven Services that support business processes is in essence an Agile approach. In fact, we have reinterpreted all four Agile principles to apply to the SOA context:

• Business-driven applications above the Service abstraction — One essential role of the Business Service abstraction at the heart of SOA is to enable business process-centric compositions of Services that both empower the business and enable business agility. This principle reinterprets the customer collaboration part of the Agile Manifesto to focus on business interactions above the Service abstraction.
• Architecture-driven iterative approach — taking an iterative approach is a long-established technique for dealing with unclear or changing business requirements. It’s no wonder, then, that all Agile methodologies are iterative. It is also essential for organizations to take an iterative approach to their SOA initiatives as well.
• Governance-driven reuse — The Service reuse business driver for SOA is essentially an Agile principle, because it focuses on leveraging software to meet diverse, changing requirements. Governance has a special role here, in that appropriately governed reuse empowers the business, enabling organizations to maximize the value they get out of their IT assets. 
• Metadata-driven development — Documentation and Service metadata are related, in that they both represent artifacts that perform important roles in the Service lifecycle. There is an important difference, however — software is for human consumption, whereas metadata are essentially for machine consumption. In other words, the focus on metadata is on driving working software. The more metadata-like your documentation is, the more you are following the Agile principle of working software over comprehensive documentation.

Agile EA

As we explained in our ZapFlash about frameworks, methodologies, and styles, SOA is a style of EA, and bringing together EA frameworks, SOA best practices, and a methodology that puts theory into practice is all essential for achieving success with your SOA initiatives. Just as EA frameworks like TOGAF are compatible with SOA, and EA styles like SOA are compatible with Agile methodologies, there’s no reason to think that EA frameworks are necessarily at odds with the Agile approach. It doesn’t make sense to say offhand that TOGAF cannot or should not be Agile. Rather, it makes sense to learn the best lessons from each approach.

What of the most significant characteristics of TOGAF is that a lot of it boils down to common sense. Much of Agile is common sense as well. The fact that documentation is a means to an end is obvious advice when you get right down to it. So why wouldn’t you follow the “documentation is a means to an end” principle with TOGAF as well? Classifications of best practices, whether they be methodologies (Agile), frameworks (TOGAF), or architectural styles (SOA), help us to some extent when it comes time to communicate and learn such practices. But you don’t want to make the mistake of being too dogmatic about any of them.

There’s no guarantee, of course, that simply because you’re adopting TOGAF that you’ll actually get the architecture right. But then again, the same can be said about SOA or Agile or any other approach that bills itself as consisting of best practices. Just because you have best practices doesn’t mean you’re going to get them right. As we like to say, you can have a diet book and read it in front of the TV while you’re eating ice cream and you won’t lose any weight!

The ZapThink Take

It’s our perspective that architecture in general is a rather loose collection of best practices that require the expertise and experience of the architecture team to know when it’s appropriate to apply them. It’s more important to apply the right practice in a given situation than to know what bucket that practice technically falls into. So it doesn’t really matter if what you’re doing is called Agile or TOGAF or SOA. As long as what you’re doing is the best practice approach for solving the problems at hand, then you’re taking the right approach.

One of the common risks of excessive architectural dogmatism is the “analysis paralysis” problem. It hardly makes sense to spend too much time on governance too early on, for example, before you have anything to govern. That being said, you need some governance in every iteration. The core best practice here is to take an iterative approach, to governance as well as most everything else. That’s the key to avoiding analysis paralysis, and in general to dealing with unclear, incompletely defined, or shifting business requirements. That’s why iterative approaches are explicitly part of both Agile and TOGAF, and also a core part of SOA.

So, this iterative principle would apply to EA as a whole as well. Don’t assume that the best approach is to tackle any particular approach to EA “in toto.” Rather, take an iterative approach to identifying those elements in the framework that apply to your situation, feel free to mix and match practices from different areas, and with each new iteration, focus ruthlessly on what works and what doesn’t. If you follow this last bit of advice, then you can rest assured you’re practicing Agile EA.