Get certified - Transform your world of work today

Negotiating Scrum Through a Waterfall

How One CSP Navigates the Difficult Waters of Implementing Scrum

10/20/2010 by Phil Southward

Negotiating a Scrum team through the often laborious processes required in a waterfall-based organisation without getting stuck between departmental snags or ensnared in the inexorable documentation eddies along the way is an arduous and, if I may be excused the pun, agile undertaking. Sitting out the umpteenth interdepartmental project sign-off meeting, a Scrum practitioner could be forgiven for thinking that the waterfall manifesto, if there was such a thing, would value:

•    Processes and tools over individuals and interactions.
•    Comprehensive documentation over working software.
•    Contract negotiation over customer collaboration.
•    Following a plan over responding to change.

In this clash of cultures, conflicts inevitably arise, yet if my experience in a couple of large (over 4,000 employees) waterfall-based companies is anything to go by, the two can be made to fit together. Somehow.

First up. Work around the keepers of the waterfall manifesto by negotiating a change to the contents of the waterfall deliverables to make them more Scrum-like. Second. Alter the Scrum artefacts and definitions to fit with the waterfall world negotiated above. Finally, as a last resort when nothing else is possible, create all the required waterfall process documentation and deliverables, while running projects internally using Scrum. Successfully negotiating the waterfall will involve some combination of all three.

Working Around the Keepers

Negotiating a change to waterfall deliverables to make them more Scrum-like is a compromise. A compromise of both the organisation waterfall process and Scrum.

How little waterfall and how much Scrum is acceptable will vary from organisation to organisation. When negotiating, try to avoid a direct confrontation over the benefits of Scrum versus waterfall. Lazy, undisciplined groups that don't follow process, do documentation, fix scope or have project plans will never win a war of words with the keepers. Instead endeavour to match Scrum artefacts and outputs as closely as possible to the deliverables required by the organisation's waterfall process, minimising unnecessary and duplicate documentation wherever possible. Aim for a continuous improvement and integration of the amount of Scrum within the waterfall process. With succeeding projects, use the success of previous projects, to renegotiate the contents of the waterfall deliverables to be less and less waterfall and more and more Scrum.

Fitting with the Wonderful Waterfall World

Whatever the outcome of negotiations with the keepers, some aspects of Scrum will need to be changed to fit in a waterfall world. Changing Scrum to fit is easy. The hard part is to ensure the changes made do not go so far that any resemblance to Scrum, and therefore the significant benefits of using Scrum, is lost. Try to find a way to adapt Scrum without throwing out what makes Scrum Scrum.

For example, there's not much benefit to using Scrum when the sprint backlog is constantly altered during the testing phase to fix the most urgent bugs. This just leads to chaos. Instead try shorter sprints (we found one week worked quite well). That'll give the team the chance to complete bug fixes they begin and therefore increase the bug resolution rate. And where is the agility (and resultant benefits) in creating a Scrum-like functional requirements document version of the product backlog if it contains every last detail of the wished for functionality months before implementation? More agile, and usually acceptable in most organisations, is to break down the larger "epic" user stories far enough an acceptance test can be added to the user story. Fill in the final detail, add further tests and/or split the user stories further just before implementation in the normal agile way.

A Last Resort

Finally if the keepers refuse to compromise, then there is not much in the way of other options than to supply them with the documentation they require while also creating a product backlog with user stories, etc. Create cross references to the waterfall documentation to ensure nothing is overseen. For example to ensure all requirements are covered by user stories in the product backlog, add an extra column to the product backlog with the requirement number.

As may be expected, this dual approach is not ideal. It requires a significant amount of additional, duplicate effort and is not exactly the most agile of method of implementing software available. It does allow Scrum to gain a slow foothold however. With a number of successful Scrum projects, the reluctance of the keepers to compromise will reduce. Eventually. Before they know it, the organisation will be implementing Scrum around them.

In The Real World

This all sounds very nice in theory, but what does it mean in practice? I assume a basic understanding of both Scrum and the waterfall software development life cycle (SDLC) phases of

•    Initiation;
•    Planning;
•    Analysis;
•    Design;
•    Development; and
•    Testing.

In this section, I outline a few practical examples on how to negotiate Scrum through a waterfall world using the methods discussed above.


Typical Waterfall Deliverable Business case for project approval/denial
Working Around the Keepers
  • Assist the project initiator to construct a business case from the combined estimates of business value and cost of each high level project feature ("epic" user story). Such a business case is easier to get through a waterfall approval process since lower value, higher cost features (user stories) maybe removed with minimal effort and the whole project split into smaller sub-projects (something financial controllers are usually found of).
Fitting In 
  • A standard business case with no breakdown into value per feature (user story). User stories are prioritised according to value, however there is no value assigned to each user story, only to the complete business case.
Last Resort
  • A standard business case with no prioritisation of user stories by value. Prioritisation takes place simply on the most "important" of the user stories, not the most valuable.



Typical Waterfall Deliverable Project plan
Working Around the Keepers
  • Supply a project plan that is really an early Scrum release plan for the project, containing the estimated number of sprints, scrum team composition, an initial velocity, estimated end date based on velocity, etc.
  • Update the release plan in the normal "Scrum way". That is continually after each sprint as usual in scrum.
  • Call the releases within the release plan milestones.
  • The test plan (also called test concept or strategy) is based on the various levels of testing both manual and automatic (e.g. unit tests, load tests, acceptance tests, integration tests, etc) used within the Sprints.
Fitting In 
  • Do the same as above for "Working Around the Keepers".
  • Allow for a series of bug fixing sprints in the waterfall test phase.
Last Resort
  • Create a traditional project plan in MS Project or similar and a Scrum release plan and update each as the project progresses.



Typical Waterfall Deliverable Functional Requirements Document
Working Around the Keepers
  • Create a functional requirements document that consists of user stories from the Product Backlog instead of a list of traditional requirements and copy or preferably link) them as a table or list into a functional requirements document "shell".
  • Split "epic" user stories just far enough an acceptance test can be added to the user story. Fill in the final detail, add further tests and/or split the user stories further just before implementation in the normal agile way.
  • Where more text is required, write simple summaries.
Fitting In 
  • If the requirements are presented as a finished list from another group (e.g. from an analysis group), create user stories as normal and map them to the existing requirements in the list by creating a new column in the Product Backlog that contains the requirement numbers covered by the user story.
  • If use cases are used, then substitute the use cases for user stories in the Product Backlog and add additional elements to the use cases (such as acceptance criteria/tests and discussions, etc) in another column to make them more like user stories.
Last Resort
  • Write a waterfall functional requirements document AND create a Product Backlog of user stories mapped to the requirements by an extra column in the Product Backlog



Typical Waterfall Deliverable Design document
Working Around the Keepers
  • As per standard Scrum, begin development during the design phase with a few short sprints to build the most important stories and verifying the proposed design. However as an additional task or set of tasks, create a design document that outlines the design agreed upon.
  • Keep the detail in the design document to the minimum required for understanding.
Fitting In 
  • As per standard Scrum, begin development during the design phase with a few short sprints to build the most important stories and test the proposed design.
  • Add a task or tasks to each user story to create and expand a design document.
  • Keep the detail in the design document to the minimum required for understanding.
  • The design document is required for the user stories to be "done".
Last Resort
  • Create a complete design document in the normal waterfall way before beginning any development. 
  • Once development begins concentrate on building the most important stories and verifying the design. 
  • If changes are required to the design (for example if problems were found in the first sprint or sprints), create an urgent "change request" and alter the design detailed in the design document.



Typical Waterfall Deliverable Code (often minimally tested)
Working Around the Keepers
  • As per standard Scrum - tested and potentially shippable code.
Fitting In 
  • Reduce potential bugs by integrating with other systems during successive sprints (as per normal agile practice). If this is not possible, build "mocks" that use data from the other systems and behave as per those systems.
  • Obtain a copy of the test cases for each feature (user story) to be used by the testing group and build and run (automate) these tests with each sprint.
  • Rather than being 100% bug free, a user story is considered "done" when it is deployed to the systems of the the organisation test group.
  • Don't just use mock objects of other organisation systems. Link/integrate to organisation wide test systems instead if possible. If this is not possible, obtain data and expected behaviour from those test systems and build into the mocks.
Last Resort
  • There is none really. Work as per the "Working Around the Keepers" and "Fitting In" sections above



Typical Waterfall Deliverable

Tested code

Bugs found in testing are fixed

Working Around the Keepers
  • As per standard Scrum - tested and potentially shipable code.
Fitting In 
  • Run shorter sprints (e.g. one week Sprints) to allow a faster resolution of urgent bugs and go give the Scrum Team the opportunity to complete bug fixes before they start on the next. That will also have the side effect of increasing the bug resolution rate.
  • Bugs found by the testing group in integration testing become new entries in the Product Backlog and may be fixed in a series of bug fixing sprints allowed for in the project plan (release plan) supplied in the planning phase.
  • For each new bug found, write an automatic test to reproduce the bug. This test is used to check the bug is properly fixed and also to ensure the bug will not reoccur.
  • Sprints during the test phase are to work only on bugs found.
Last Resort
  • There is none really. Work as per the "Working Around the Keepers" and "Fitting In" sections above

Summing Up

Working around those sticking resolutely to the waterfall process, altering Scrum (just a little bit), or even duplicating effort by running Scrum and a waterfall SDLC side by side for the same project, are not ideal solutions to the problems of negotiating a Scrum team through a waterfall-based organisation. Although they worked for us, others will likely find superior, more suitable methods than those outlined here. And that is good. Continuous improvement is essential. The keepers never sleep.