Can Agile Really Work in Heavily Regulated IT Environments?
For the past several decades, Waterfall has been the leading software development methodology in heavily regulated IT environments. However, Waterfall’s popularity has declined in recent years as heavily regulated industries have increased acceptance of Agile software development practices. For example, the Federal government has released a number of recent publications and policy initiatives encouraging the use of Agile development practices and is establishing a new contract vehicle for Agile delivery services 1. The gradual emergence of Agile methodologies is encouraging for IT service providers looking for opportunities to leverage iterative development and delivery strategies to propel heavily regulated IT industries to the next level of efficiency.
Waterfall vs. Agile
There are a number of reasons that the government and other heavily regulated industries, such as health care, life sciences, energy and utilities, are shifting to Agile software development practices, but the major reason is likely the very low success rates2 that these industries have experienced with Waterfall as the dominant software development methodology.
Waterfall projects in heavily regulated IT environments often fail due to the model’s demand that requirements must be completely specified and considered final in the beginning of the project (or else undergo a rigid change control process). However, as all IT stakeholders know, evolving requirements are a standard feature of any software development project, and they are particularly the norm for industries with complex mission and business needs. Thus, it is no surprise that the mismatch that results from Waterfall’s process (fixed requirements) and what actually happens (requirements evolve) typically yields systems and software products that do not meet end-user needs and fail in acceptance or adoption. With these consistently sub-par outcomes, Waterfall projects are especially vulnerable to cost and schedule overruns.
Agile methodologies, on the other hand, are singularly focused on creating software that meets end-user needs. These methodologies attempt to tackle many of the common problems experienced by Waterfall projects through a hands-on approach to requirements gathering and development. Agile methodologies approach requirements gathering by encouraging constant stakeholder communication and collaboration throughout the course of the project as opposed to defining all requirements up front. This continuous engagement brings the requirements definition and development processes closer together and helps stakeholder identify changes in the requirements early and often. Thus, by constantly reexamining customer needs, Agile increases the opportunity for project teams to succeed in delivering functionality that actually meets end user needs and helps achieve end-user adoption.
While many studies have found Agile methodologies to be more successful than Waterfall overall 3, Agile is not a silver bullet for solving every project’s cost and schedule overruns. Transitioning a program in heavily regulated IT environments from Waterfall to Agile presents very tough challenges. For example, structured and regulated environments often adhere to stage gate reviews and have significant documentation requirements that may seem unnecessary to many Agile practitioners. These challenges are compounded by the fact that many of the IT systems in these environments are mission-critical and cannot experience major downtime. So how can IT service providers reap the benefits of Agile development methodologies for their customers in these heavily regulated environments?
The first step to implementing Agile in heavily regulated environments involves initiating a cultural shift and helping stakeholders embrace change. Many existing programs in these industries were built on very rigid and formally documented processes that align with the Waterfall model, and as a result, these industries are often skeptical of Agile methodologies that tout their “flexible” approaches. Therefore, IT services providers transitioning a program in one of these heavily regulated industries must address the common misconceptions of Agile as an ad-hoc, unstructured process and show their stakeholders what Agile really is – a formal, disciplined, and established process that produces quality outcomes in an environment with evolving requirements. This first step is critical as stakeholders are very unlikely to consider, let alone embrace, a transition to Agile until they understand the basic principles of Agile.
Since the demands of Agile may seem intimidating to early adopters, we recommend first identifying projects that are good candidates for early adoption. Projects which are suitable for early adoption are typically smaller in size and have active, engaged project sponsors. Once a project has been identified, service providers can create a project roadmap for transitioning from Waterfall to Agile and work together with customers to build a plan that details the transition for each major process area. Sound discipline and collaboration are critical to this step as well as creating a culture of success. We recommend creating a culture of success by starting with small changes and early “wins” to further engage stakeholders and increase morale and then gradually moving towards more complex tasks to further demonstrate Agile’s benefits. Then, once the team demonstrates consecutive successes, Agile’s transparency and efficiencies will become the impetus for full acceptance.
We employed the techniques mentioned above in 2011 when a customer group at the Food and Drug Administration (FDA) asked us to transition a project from Waterfall to Agile. The customer group, which was one of five within an enterprise-wide, mission-critical program, needed to deliver functionality to production quickly, and the typical six-month Waterfall timeframe was not fast enough to meet Congressionally mandated deadlines. Due to the project’s narrow scope and the project sponsor’s high level of engagement, our team identified the project as a suitable candidate for transitioning to Agile. We developed a roadmap for the project and introduced a gradual turn from Waterfall terminology, artifacts, and processes to their Agile counterparts. Our team then began development on the high priority, low level of complexity items to demonstrate progress and success to our customers before moving onto the more complex items. In the end, we successfully delivered all of the functionality by the Congressionally mandated deadlines, and the customer group elected to continue using Agile over Waterfall for future projects. Additionally, within a year of the first group’s successful transition, all of the other customer groups eventually transitioned to Agile, and our speed to market for the program was accelerated by four to six months.
1 – https://18f.gsa.gov/
2 – http://www.infoq.com/resource/articles/scaling-software-agility/en/resources/ch02.pdf
3 – http://www.ambysoft.com/surveys/success2013.html