Scrum Buts, “Meta” Thinking, and Agile Architecture

ZapThink has long bemoaned the Agile Manifesto paradox: that the point to the Manifesto was to be less dogmatic about software development, but today people are overly dogmatic about Agile, defeating its entire purpose. In fact, this paradox has found its way into what is perhaps the most popular of the Agile methodologies: Scrum. Not to worry, all you Scrum aficionados out there; we’re not going to teach you how to do Scrum in this ZapFlash. Instead, this article is more about how to think about a broad set of problems in a particular way, starting with Scrum Buts.

Enter the Scrum But

The notion of a Scrum But arose when it became clear that thousands of organizations were attempting to follow Scrum for their software development projects, but many of them were having problems with one or another of its tenets. As a result, they would say things like:

“We use Scrum, but Retrospectives are a waste of time, so we don’t do them.”

Or:

“We use Scrum, but we can’t build a piece of functionality in a month, so our Sprints are 6 weeks long.”

Where Retrospectives and Sprints are well-known Scrum best practices. Note that both of these statements follow the same “We use Scrum, but X so Y” pattern, hence the term Scrum But.

The question with Scrum Buts, of course, is what to do with them—or more specifically, how to think about them. There are two schools of thought:

1. Scrum Buts are simply excuses not to do Scrum properly, and if you’re not doing it properly, then you’re not really doing it at all.
2. Resolving Scrum Buts when they come up in order to achieve the desired result (software that meets the stakeholders’ needs) is actually a part of the Scrum methodology. As a result, Scrum Buts are expected and even welcomed.

From ZapThink’s perspective, option #1 is an example of taking a dogmatic approach, which is inherently non-Agile. Option #2 basically says that you can modify the rules if necessary (one of the four principles of the Agile Manifesto), even the rules of Scrum itself.

In other words, principle #2 is self-referential—which may be more Agile to be sure, but people have problems with self-reference. It brings up uncomfortable visions of the liar’s paradox: “everything I say is a lie,” or in more modern terms, the first rule of Fight Club (“do not talk about Fight Club.”) How can we make sense of the world or anything in it if we have to deal with self-reference paradoxes? If a rule of Scrum is “you can change the rules of Scrum,” then couldn’t anything and everything be Scrum? What use is that?

“Meta” Thinking

Fortunately, such problems of self-reference have a straightforward, if subtle solution. Instead of thinking of such statements as referring to themselves, think of them as actually two separate but related statements, where one relates to the other. We call this “meta” thinking.

In the case of Scrum Buts, we have the Scrum methodology and the Scrum meta-methodology. A meta-methodology is a methodology for creating methodologies. Remember, the Scrum methodology is a methodology for creating software. The Scrum meta-methodology is a methodology for creating or improving the Scrum methodology. So when someone says:

“When you say, ‘We use Scrum, but we can’t build a piece of functionality in a month, so our Sprints are 6 weeks long,’ my recommendation is to try three 30-day Sprints before extending the length of the Sprint.”

That entire statement is a Scrum meta-methodology statement. Furthermore, without such statements, your methodology wouldn’t be Agile. The obvious conclusion is that all Agile methodologies are actually meta-methodologies. Otherwise they wouldn’t be Agile!

Avoiding the Hall of Mirrors

I’m sure one of you wise guys out there is thinking, what about methodologies for creating meta-methodologies? We’d call meta-meta-methodologies, of course. And what about methodologies for creating those, ad infinitum? We call this the hall of mirrors problem, because you only need to have two mirrors facing each other to get the infinite tunnel effect.

Simple answer: we don’t need a methodology for creating meta-methodologies. Instead, an informal approach will do. In general, we only want to go to the meta-meta step if there’s a bona fide reason to do so, as with Model-Driven Architecture, when they talk about meta-meta-models. But even the brainiacs at the OMG don’t spend much time thinking about meta-meta-meta-models—at least, I hope not!

We often get this question in our Licensed ZapThink Architect SOA course when we discuss meta-policies. A meta-policy, of course, is a policy that applies to other policies. Because the automation of policy-related processes is a core enabler of SOA governance, polices for how an organization performs such automation are quite important—and such policies are meta-policies. An example of a useful meta-policy: “we’ll configure our registry/repository’s design-time workflow to conform to our existing Service lifecycle.” Such a configuration means configuring the policies in the tool as a way of automating the corresponding workflow, and thus that statement is a meta-policy.

In this example, the hall of mirrors question focuses more on automation then on the policies themselves. Does it make sense to automate meta-policy related processes? The answer is basically no. It’s best to create, communicate, and enforce meta-policies manually, through human communication activities. Yes, you could think of that last statement as a meta-meta-policy, but thinking about it that way doesn’t really help you, so don’t bother.

More Meta-Thinking

Meta-models are well known now, and other than those plus meta-methodologies and meta-policies, where does meta thinking come up? Way back in 2005, ZapThink discussed the notion of meta-processes. A meta-process, of course, is a process for dealing with processes, for example, the process of composing Services to implement a business process. While we pointed out in that article that such meta-processes should be handled manually, we also predicted increasing automation of Service composition processes—a capability that is still well in the future (although if you have any examples, please let us know.)

Another meta that is central to ZapThink’s thinking is the meta-requirement. In particular, we’ve been talking about the meta-requirement of business agility as a fundamental driver of SOA, and by extension, Agile Architecture in general. When the business asks for an agile system, they are asking for a system that can respond to changing requirements—which is what makes such agility a meta-requirement. And remember, it’s because the business agility meta-requirement is an emergent property of the enterprise that requires us to characterize the enterprise itself as a complex system.

Finally, at the risk of belaboring the point, let’s talk about meta-architecture: what does it mean to architect an architecture? Yes, ZapThink has been spending a lot of our brain cycles on meta-architecture as well. We’ve been putting our stamp on how best to do SOA for several years now. Our students may be architecting their organizations and their component systems, but ZapThink has been architecting SOA itself. And now that we can stick a fork in that, it’s time to work on architecting Cloud architecture and more broadly, Agile Architecture.

The ZapThink Take

Meta thinking about something means more than just thinking about the thing, it means thinking about how the thing might change. Meta-processes are processes for changing processes. The reason we need meta-policies is because policies might change—otherwise we’d just hard-code them into our software and be done with them. The business agility meta-requirement is fundamentally a requirement for change.

Complex systems are themselves self-adapting. They are always in a state of change. How would we ever expect to architect a complex system like an enterprise if we didn’t take an Agile Architecture approach that worked at the meta level to deal with change? Perhaps the lack of a complex systems approach to traditional Enterprise Architecture—architectural approaches that presume a final to-be state—is why all such approaches are inherently flawed.

Our Scrum But example is an illuminating illustration of what we mean by an Agile Architectural approach. You could look at a list of Scrum Buts and say, this team is doomed to failure. They’ve taken the good bits to Scrum and stripped those out, and now they’re screwed. Alternatively, you could look at the same list and say, with a few bits of advice about how to deal with the Scrum Buts (in other words, the right meta-methodology), this team might be successful after all.

This admittedly oversimplified scenario has two outcomes: team crashes and burns, or team is successful in spite of their Scrum Buts. In complex systems theory, these outcomes are called attractors: given a set of circumstances, the end result will usually follow one of a set of patterns, in spite of the fact that different people are involved, each with their own skills and preferences. The system is subject to perturbations (the Scrum Buts, in our example), as well as constraints (the advice that makes up the meta-methodology), and the various identities of the people.

Without the appropriate advice, the attractor that is most likely to describe the outcome is the failure attractor. Clearly, the more Scrum Buts you have, and the more serious they are, the more likely your project will fail (although failure is still not certain). But with the proper meta-methodology, you can steer the project toward the success attractor, in spite of all the Scrum Buts and the various people on the team, who may be disgruntled, incompetent, overworked, or whatever.

Note that the success attractor is not a final state in a traditional sense. Rather, it allows for the fact that perturbations, constraints, and identities are always subject to change. Generalize our Scrum meta-methodology example to the level of the enterprise, and you can get a sense of what we mean by Agile Architecture. Can we design the complex system of the enterprise, a system consisting of human and technology subsystems, to move toward desirable attractors through the introduction of appropriate meta-policies, meta-processes, and meta-methodologies? That’s the million dollar meta-architecture question.