This article is based on my observations while working with large U.S. financial organizations in the Midwest. It documents my personal observations on the reluctance of large and complex organizations to take their first step toward Agility.
When it comes to being Agile, people are naturally reluctant to change. They demand enough evidence that Agile will work for them, especially in the current economy, in which no one wants to risk his or her job. Why would you put everything on the line for some flashy Agile thing? "Show me the proof that it works," they say. These organizations (especially financial institutions) are already heavily constrained by regulations imposed by financial regulatory authorities, so one wrong move could jeopardize their business and reputation in the marketplace.
The first step
Considering their reluctance, I think people are more open to frequent small changes than one big change. When they take their first baby step toward agility, they try an iterative approach only for a portion of the project (somewhere in the middle), while they still want the overall project to be mostly predictable and plan driven.
What does Agile delivery mean?
Although there are many definitions of Agile delivery, this is one that I personally like: It's about creating a quality product that provides high business value while reducing the risk of failure.
And when I talk about Agile, I look at it from a broader perspective, instead of getting hung up on the semantics or any preconceived notion.
How is Water-Scrum-Fall implemented?
Although not ideal, I think that the Water-Scrum-Fall approach is not a bad start for these organizations, as long as they understand its limitations. On one side, Water-Scrum-Fall can be misleading for people who haven't tried Agile before and are skeptical about change and transformation. On the other side of the spectrum, I've seen organizations that started with the Water-Scrum-Fall approach, realized the benefits, and then started capturing data to measure their success. The metrics captured were usually in terms of savings in manpower cost, risk reduction, and reduction in defects or cost of production.
At a high level, a typical project goes through the Initiate, Plan/Discover, Build, and Deploy/Release phases, which are heavily managed by respective control gates. With Scrum being the most popular of many Agile frameworks available today, when we talk about Water-Scrum-Fall, these organizations usually introduce Scrum in the middle of their delivery process (the Build phase of the project), and as they mature, they expand on either side into other phases of the project.
Water-Scrum-Fall project life cycle
As a result, in such organizations (that are new to this practice), the upstream activities are executed by using the Waterfall methods in which they create plan-driven business cases, budgeting, and requirements, to the extent that they think they have a sufficient level of detail and a good degree of confidence in the project. These requirements (in the form of a product backlog) are then traditionally thrown over the fence to the delivery team. Finally, these delivery teams start working in a timeboxed, iterative fashion, which is a bit more Scrum-like.
Furthermore, once the development team completes the Build phase activities, the product (which is close to release ready) is handed over to the dedicated system integration testing team. In a more plan-driven Waterfall way, this team then executes their manual test scripts to mitigate whatever risk might have been introduced due to the iterative nature of development in the middle of the project life cycle. This is where the project switches back to Waterfall mode to carry out manual system integration testing and other remaining project-specific activities (e.g., security scans, manual regression, UAT, and deployment-related tasks). This is the final phase of the project that I refer to as the "Fall" in the Water-Scrum-Fall hybrid approach.
Why do organizations embrace Water-Scrum-Fall?
I can see why these large, complex organizations are driving the behavior of Water-Scrum-Fall. If you think about if from their standpoint, you certainly would like to see some level of detail in the initial phases of the project. These details ensure the success of these projects or programs and at the same time provide them the value they want.
Another reason for thinking that they do the initial phase of the project in a plan-driven way in Waterfall is that it's difficult for sponsors or executive management to make that mental leap of faith. It's difficult to say, "I'm giving you hundreds of thousands of dollars, so how about you try out the new Agile approach on this project and see if it works for us? And if it does work, I'll give you another million dollars to finish the project." But what if the Agile way doesn't work? They want that sense of security up front.
When it comes to adopting Agile principles and practices, what I've observed is that when you come in the middle phase of the project, the developers and testers actually love being Agile. Even independent studies have proven that these roles are the quickest to adopt and adapt to new practices. The reasons are that it is a lightweight framework, and it actually empowers them and gives them accountability and opportunities to collaborate. So why not?
But for people involved in upstream and downstream project phases (i.e., project phases other than the Build phase), I haven't seen them invest in their engineering practices, such as continuous integration, automation, refactoring, and figuring out how they can push the code a lot quicker through the final testing and deployment stages. People don't realize that investing in these activities and planning for them in each sprint will help them mitigate the project risk up front.
Take it to the next level -- how did we do it?
In my organization (a bank), after maturing in our Water-Scrum-Fall practices and attaining a sustainable pace of delivery, we decided to inject the Planning-Discover phase of the project with Agile-Scrum principles. We prepared a business case (supplemented with key metrics and projected savings) and presented it to senior management, business partners (product managers), the enterprise architecture review board, the practice area lead for business analysis, the user experience group (UX), and the enterprise project management office (EPMO), with recommendations to pilot an iterative approach in the Planning-Discover phase.
The key to the success of this pilot was in embracing three things:
Collaboration between business and IT partners
Identification of minimum marketable features (MMFs)
Early and continuous prioritization of the product backlog
Having a this-is-good-enough mindset
After the pilot was approved and before project kickoff, we tried to embed these three values through the appropriate level of training and coaching for some of the key roles (PO/BA, development team, and project managers) that were going to be affected by this change. The goal was to make sure that everyone understood their role and also to ensure that the PMO was comfortable with revising their guidelines so that the project managers could execute the Plan-Discover and Build phases of the project in an overlapping or iterative fashion.
As a result, once MMFs were identified and prioritized, we focused on the product backlog and defined the initial set of user stories with sufficient details from the requirements, design and architecture, UX framework, and wireframes standpoints. As these MMFs were fleshed out and made ready for development sprints, the development team pulled the product backlog items (PBI) into their sprints while product owners were working on the next set of MMFs from the prioritized backlog.
The pilot was a success. We realized that it's OK to be Waterfall in the initial phase when the business case is prepared and the initial funds are procured. However, as soon as we started talking about the solution and its associated requirements, we needed to be bit more Agile. We created a pipeline of work for the standing Agile teams while we worked in parallel on other features rather than waiting for all the requirements to be completed up front.
As a next step, we want to revise our Definition of Done and eliminate the concept of a hardening sprint. Our recommendation would be to seek opportunities in identifying and creating PBIs and tasks today that are specifically implemented in the hardening sprint. Work with the PO to identify and prioritize these tasks in the product backlog and then start pulling these tasks in relevant development sprints. These tasks are typically around system integration testing, overall regression testing, fixing issues reported by application security scans, and so on.
If you truly embrace the Agile Manifesto and its guiding principles, and use the right framework (Scrum, Kanban, XP, and any other frameworks) tailored to your needs, with even a little bit of the plan-driven approach, I'm sure this will certainly make a positive difference, provided you do it the right way.
Humble J, Molesky J, O'Reilly B. Lean Enterprise.
O'Reilly Media, Inc. December 2014.