Negotiating Scrum Through a Waterfall

How One CSP Navigates the Difficult Waters of Implementing Scrum

20 October 2010

Negotiating a Scrum team through the often laborious processes required in a waterfall-based organisation without getting stuck between departmental snags or ensnared in the inexorable documentation eddies along the way is an arduous and, if I may be excused the pun, agile undertaking. Sitting out the umpteenth interdepartmental project sign-off meeting, a Scrum practitioner could be forgiven for thinking that the waterfall manifesto, if there was such a thing, would value:

•    Processes and tools over individuals and interactions.
•    Comprehensive documentation over working software.
•    Contract negotiation over customer collaboration.
•    Following a plan over responding to change.

In this clash of cultures, conflicts inevitably arise, yet if my experience in a couple of large (over 4,000 employees) waterfall-based companies is anything to go by, the two can be made to fit together. Somehow.

First up. Work around the keepers of the waterfall manifesto by negotiating a change to the contents of the waterfall deliverables to make them more Scrum-like. Second. Alter the Scrum artefacts and definitions to fit with the waterfall world negotiated above. Finally, as a last resort when nothing else is possible, create all the required waterfall process documentation and deliverables, while running projects internally using Scrum. Successfully negotiating the waterfall will involve some combination of all three.

Working Around the Keepers

Negotiating a change to waterfall deliverables to make them more Scrum-like is a compromise. A compromise of both the organisation waterfall process and Scrum.

How little waterfall and how much Scrum is acceptable will vary from organisation to organisation. When negotiating, try to avoid a direct confrontation over the benefits of Scrum versus waterfall. Lazy, undisciplined groups that don't follow process, do documentation, fix scope or have project plans will never win a war of words with the keepers. Instead endeavour to match Scrum artefacts and outputs as closely as possible to the deliverables required by the organisation's waterfall process, minimising unnecessary and duplicate documentation wherever possible. Aim for a continuous improvement and integration of the amount of Scrum within the waterfall process. With succeeding projects, use the success of previous projects, to renegotiate the contents of the waterfall deliverables to be less and less waterfall and more and more Scrum.

Fitting with the Wonderful Waterfall World

Whatever the outcome of negotiations with the keepers, some aspects of Scrum will need to be changed to fit in a waterfall world. Changing Scrum to fit is easy. The hard part is to ensure the changes made do not go so far that any resemblance to Scrum, and therefore the significant benefits of using Scrum, is lost. Try to find a way to adapt Scrum without throwing out what makes Scrum Scrum.

For example, there's not much benefit to using Scrum when the sprint backlog is constantly altered during the testing phase to fix the most urgent bugs. This just leads to chaos. Instead try shorter sprints (we found one week worked quite well). That'll give the team the chance to complete bug fixes they begin and therefore increase the bug resolution rate. And where is the agility (and resultant benefits) in creating a Scrum-like functional requirements document version of the product backlog if it contains every last detail of the wished for functionality months before implementation? More agile, and usually acceptable in most organisations, is to break down the larger "epic" user stories far enough an acceptance test can be added to the user story. Fill in the final detail, add further tests and/or split the user stories further just before implementation in the normal agile way.

A Last Resort

Finally if the keepers refuse to compromise, then there is not much in the way of other options than to supply them with the documentation they require while also creating a product backlog with user stories, etc. Create cross references to the waterfall documentation to ensure nothing is overseen. For example to ensure all requirements are covered by user stories in the product backlog, add an extra column to the product backlog with the requirement number.

As may be expected, this dual approach is not ideal. It requires a significant amount of additional, duplicate effort and is not exactly the most agile of method of implementing software available. It does allow Scrum to gain a slow foothold however. With a number of successful Scrum projects, the reluctance of the keepers to compromise will reduce. Eventually. Before they know it, the organisation will be implementing Scrum around them.

In The Real World

This all sounds very nice in theory, but what does it mean in practice? I assume a basic understanding of both Scrum and the waterfall software development life cycle (SDLC) phases of

•    Initiation;
•    Planning;
•    Analysis;
•    Design;
•    Development; and
•    Testing.

In this section, I outline a few practical examples on how to negotiate Scrum through a waterfall world using the methods discussed above.

Initiation

Typical Waterfall Deliverable Business case for project approval/denial
Working Around the Keepers
  • Assist the project initiator to construct a business case from the combined estimates of business value and cost of each high level project feature ("epic" user story). Such a business case is easier to get through a waterfall approval process since lower value, higher cost features (user stories) maybe removed with minimal effort and the whole project split into smaller sub-projects (something financial controllers are usually found of).
Fitting In 
  • A standard business case with no breakdown into value per feature (user story). User stories are prioritised according to value, however there is no value assigned to each user story, only to the complete business case.
Last Resort
  • A standard business case with no prioritisation of user stories by value. Prioritisation takes place simply on the most "important" of the user stories, not the most valuable.

 

Planning

Typical Waterfall Deliverable Project plan
Working Around the Keepers
  • Supply a project plan that is really an early Scrum release plan for the project, containing the estimated number of sprints, scrum team composition, an initial velocity, estimated end date based on velocity, etc.
  • Update the release plan in the normal "Scrum way". That is continually after each sprint as usual in scrum.
  • Call the releases within the release plan milestones.
  • The test plan (also called test concept or strategy) is based on the various levels of testing both manual and automatic (e.g. unit tests, load tests, acceptance tests, integration tests, etc) used within the Sprints.
Fitting In 
  • Do the same as above for "Working Around the Keepers".
  • Allow for a series of bug fixing sprints in the waterfall test phase.
Last Resort
  • Create a traditional project plan in MS Project or similar and a Scrum release plan and update each as the project progresses.

 

Analysis

Typical Waterfall Deliverable Functional Requirements Document
Working Around the Keepers
  • Create a functional requirements document that consists of user stories from the Product Backlog instead of a list of traditional requirements and copy or preferably link) them as a table or list into a functional requirements document "shell".
  • Split "epic" user stories just far enough an acceptance test can be added to the user story. Fill in the final detail, add further tests and/or split the user stories further just before implementation in the normal agile way.
  • Where more text is required, write simple summaries.
Fitting In 
  • If the requirements are presented as a finished list from another group (e.g. from an analysis group), create user stories as normal and map them to the existing requirements in the list by creating a new column in the Product Backlog that contains the requirement numbers covered by the user story.
  • If use cases are used, then substitute the use cases for user stories in the Product Backlog and add additional elements to the use cases (such as acceptance criteria/tests and discussions, etc) in another column to make them more like user stories.
Last Resort
  • Write a waterfall functional requirements document AND create a Product Backlog of user stories mapped to the requirements by an extra column in the Product Backlog

 

Design

Typical Waterfall Deliverable Design document
Working Around the Keepers
  • As per standard Scrum, begin development during the design phase with a few short sprints to build the most important stories and verifying the proposed design. However as an additional task or set of tasks, create a design document that outlines the design agreed upon.
  • Keep the detail in the design document to the minimum required for understanding.
Fitting In 
  • As per standard Scrum, begin development during the design phase with a few short sprints to build the most important stories and test the proposed design.
  • Add a task or tasks to each user story to create and expand a design document.
  • Keep the detail in the design document to the minimum required for understanding.
  • The design document is required for the user stories to be "done".
Last Resort
  • Create a complete design document in the normal waterfall way before beginning any development. 
  • Once development begins concentrate on building the most important stories and verifying the design. 
  • If changes are required to the design (for example if problems were found in the first sprint or sprints), create an urgent "change request" and alter the design detailed in the design document.

 

Development

Typical Waterfall Deliverable Code (often minimally tested)
Working Around the Keepers
  • As per standard Scrum - tested and potentially shippable code.
Fitting In 
  • Reduce potential bugs by integrating with other systems during successive sprints (as per normal agile practice). If this is not possible, build "mocks" that use data from the other systems and behave as per those systems.
  • Obtain a copy of the test cases for each feature (user story) to be used by the testing group and build and run (automate) these tests with each sprint.
  • Rather than being 100% bug free, a user story is considered "done" when it is deployed to the systems of the the organisation test group.
  • Don't just use mock objects of other organisation systems. Link/integrate to organisation wide test systems instead if possible. If this is not possible, obtain data and expected behaviour from those test systems and build into the mocks.
Last Resort
  • There is none really. Work as per the "Working Around the Keepers" and "Fitting In" sections above

 

Testing

Typical Waterfall Deliverable

Tested code

Bugs found in testing are fixed

Working Around the Keepers
  • As per standard Scrum - tested and potentially shipable code.
Fitting In 
  • Run shorter sprints (e.g. one week Sprints) to allow a faster resolution of urgent bugs and go give the Scrum Team the opportunity to complete bug fixes before they start on the next. That will also have the side effect of increasing the bug resolution rate.
  • Bugs found by the testing group in integration testing become new entries in the Product Backlog and may be fixed in a series of bug fixing sprints allowed for in the project plan (release plan) supplied in the planning phase.
  • For each new bug found, write an automatic test to reproduce the bug. This test is used to check the bug is properly fixed and also to ensure the bug will not reoccur.
  • Sprints during the test phase are to work only on bugs found.
Last Resort
  • There is none really. Work as per the "Working Around the Keepers" and "Fitting In" sections above

Summing Up

Working around those sticking resolutely to the waterfall process, altering Scrum (just a little bit), or even duplicating effort by running Scrum and a waterfall SDLC side by side for the same project, are not ideal solutions to the problems of negotiating a Scrum team through a waterfall-based organisation. Although they worked for us, others will likely find superior, more suitable methods than those outlined here. And that is good. Continuous improvement is essential. The keepers never sleep.

Article Rating

Current rating: 0 (0 ratings)

Comments

Frank Szendzielarz, CSM, 10/21/2010 2:07:25 AM
I think the article is helpful for those who are trying to achieve such things,but I can't help feeling that it misses the core of the problem.

Organisations that are insist on unnecessary overhead, bloated process, and other seeming inefficiencies have a fundamental problem in their incentive structure. The stakeholders do not want lean processes. They want ways to maintain the status quo, keep themselves and their allies in a job and to maintain budgets.

In general, large Western organisations are not tied to manufacturing industry because their host nations no longer produce anything. Capital has flowed to the East. These Western nations have survived for the last 30 years by constant rises in household debt and rising trade imbalances with foreign exporters. While this has been happening, the West has had to keep its labour force employed while the service sector has been growing.

Without bloat and unnecessary pointless activity, how do you keep millions of office workers in a job and able to pay their over-inflated mortgages?
Willie Mora, CSM, 10/21/2010 11:41:24 AM
SCRUM has done great things for the profession or project management. However, it is sad to see so much effort directed at what is wrong with waterfall when in fact both SCRUM and Waterfall have their strong and weak points. The one important thing to remember is that no matter what methodology is being employed, a degree of the SDLC (Initiation, Planning, Analysis, Design, Development and Testing) must be completed to ensure the work is done with quality and meets user expectations. Generally, Waterfall is difficult to institute change and considers the project as a whole while Scrum is flexible and breaks the project in smaller deliverables/efforts and timelines. In reality, both Waterfall and scrum have change management procedures that allow changes. Both Waterfall and SCRUM have overrun budgets. If not implemented correctly, both SCRUM and Waterfall can be stoic and inflexible. Instead of always pitting one process over the other; the recommendation is get buy-in by focusing on how the two methodologies can co-exist in an organization to make the business more profitable. Some projects require waterfall processes because of regulatory requirements while others can and should be based on scrum processes to meet to market demands. Project Managers/Scrum Masters should be working to convince management on the benefits and profitability of a dual environment instead of pitting SCRUM against waterfall and the wall of resentment that may or may not exist.
Jan De Baere, CSP,CSPO, 10/22/2010 11:12:39 AM
I fully agree with Willie Mora. At my client ΓÇô IΓÇÖm a consultant ΓÇô I introduced scrum as an alternative to waterfall. The method used depends on the type of project and competences of the people. Condition to go scrum is the support of business and program management.

All too often I hear the opinion that when itΓÇÖs not all scrum it doesnΓÇÖt work. If that would be the case IΓÇÖm afraid scrum wouldnΓÇÖt last very long and it would only prove that scrum is not very flexible.
Phil Southward, CSP,CSM, 10/25/2010 3:23:50 AM
Thank you all for your comments. As a big fan of scrum and its benefits, I obviously would prefer to see an organisation go fully agile, rather than implement a mix of waterfall and scrum. However, in my experience, going fully agile is sometimes not possible, especially at the beginning of a scrum transition. Therefore I would like to reiterate that the point of my article was not to promote the use scrum and waterfall together, however, nor was it to pit one against the other. Rather it aimed to outline some, hopefully, helpful practical tips, gained from experience, on how to fit scrum into a waterfall environment when an organisation is unable, or unwilling, for one reason or another, to implement scrum fully.
Chris Watts, CSP,CSM, 10/26/2010 9:14:58 AM
I agree with Willie. If you look at Waterfall compared to Agile it is like comparing a railway journey to a saling trip. The railway journey is predicatble and in addition to a start and finish point has predefined stops and junctions. The sailing trip has also a start and finish point but does not always know the course in between as constant adjustment is needed. As a consultant to some of the world's largest Investment Banks, whilst they would like to take the more refreshing maritime journey, the avalanche of compliance and regulatory requirements enforces them to keep to the daily rail commute. The financial service providers see many of their projects as part of an overall investment portfolio and have invested heavily in Project and Portfolio Management tools which meet the compliance and regulatory requirements. HP PPMC, SAP, CA Clarity and Oracle PPM are good examples.
At last some of these toolsets are aligning to Agile principles so one opportunity as Scrum advocates is to promote the uptake of Agile aligned toolsets within these large corporates.
Jürgen Szolay, CSM, 11/25/2010 10:16:48 AM
First I want to say thanks to Phil for his article. Because it is helpful to get grip on using WaterScrum or Scrummerfall :)

But I want to add that my Scrum Trainer & Coach told me that there are people who just do not fit to Scrum. And you should not try to make them using Scrum at any cost.

So I guess this is the same for companies/organizations. Some will not fit to Scrum. And you should to bend and fix this agile framework to make it working within that company/organization.

Too much of compromises are a bad deal at the end.
Phil Southward, CSP,CSM, 12/5/2010 10:31:22 AM
Hi J├╝rgen,

you are dead right that some organisations/companies will just not fit to Scrum and you shouldn't try to make them. But do you know which ones those are? It is often hard to know which organisations will never adopt Scrum at the beginning. You never know if you will be surprised. Like I was when one organisation where I spent over two years negotiating scrum through their waterfall, announced Scrum was a great idea after all and they were going to transition the whole company over to Scrum and drop waterfall altogether.

You must Login or Signup to comment.