It's All in the Delivery

12 June 2006

John Hill
IconATG

Many traditional waterfall projects rely on a set of calculations called “Earned Value” and “Schedule Performance Index” to indicate whether their project is on schedule. These metrics can be misleading and often mask some of the schedule delays they are designed to predict and prevent. Scrum provides project stakeholders with more accurate expectations regarding schedule, scope and budget than traditional “Earned Value” and “Schedule Performance Index” metrics can provide for waterfall projects.Where did the metrics originate?

Where did the metrics originate?

In 1967 the Department of Defense (DOD) introduced a set of thirty-five Cost/Schedule Control System Criteria (C/SCSC). Implementing these criteria caused complex modifications to management control systems for companies doing business with the DOD. This overall “system” and its many byproducts are now generally accepted across a variety of non-military industries (“It must be valid if the military requires it”). It has become so mainstream that its logic is built into Microsoft Project and other scheduling software.

In Put Earned Value (C/SCSC) Into Your Management Control System, Quentin W. Fleming defines five metrics required for the C/SCSC system to work (BCWS, BCWP, ACWP, SV, and CV). Of these five, only the first two definitions pertain to this discussion:

  1. The Plan or Budgeted Cost for Work Scheduled (BCWS)—may be called work scheduled, which is synonymous with budget or plan.
  2. Earned Value or Budgeted Cost for Work Performed (BCWP)—may be referred to as work performed, which is synonymous with the term Earned Value.

By dividing the work performed (BCWP) by the work scheduled (BCWS) at a specific point in time, you allegedly determine how well the actual work is tracking to the scheduled work: the Schedule Peformance Index (SPI). To determine whether or not a waterfall project is “on track,” SPI is measured religiously at specified intervals (usually monthly) using this formula: SPI = (BCWP/BCWS). Any time the SPI is less than one, the schedule is deemed to be in trouble. If, for example, 5,000 hours of work had been scheduled (BCWS) for the period being evaluated and only 4,500 hours had completed (BCWP), the SPI would be .9 (4,500/5000 = .9). This means that the project is 10 percent behind schedule. If, on the other hand, 4,500 hours had been planned and 5,000 hours had been completed, the SPI would be 1.1 (5,000/4,500 = 1.1), implying that the project is 10 percent ahead of schedule.

What’s Misleading about These Metrics?

The problem with these metrics is that traditional waterfall projects include many tasks that must be completed before any software is delivered. Consider a ten-month project whose definition phase represents 30 percent of the total work. After the definition phase is finished, 30 percent of the budget has been spent and 30 percent of the project is complete, yet there is no working software. In fact, the project will continue to consume measurable costs until the software is delivered (near the planned end of the project). If the total project was budgeted at $6 million and 30 percent of the time is required for specifications, you have just spent $1.8 million (30 percent of $6 million) for a pile of specifications, meeting minutes, approval forms, memos, and so on (see Figure 1).

Figure 1

Figure 1. Ten-month (60,000-hour) waterfall project at 30 percent complete.

Another problem with these metrics is that relying on SPI and Earned Value calculations often causes the late discovery of problems in the development cycle by masking real issues until near the end of the project. Both calculations show that acceptable progress is being made over time, without ensuring that the software will even work. If significant design problems are discovered late in the project schedule, typical responses include scheduling unplanned re-design, re-coding, and retesting (usually at the expense of previously planned QA). Project managers also struggle to explain why projects are at risk late in the cycle, especially after submitting acceptable monthly SPI and Earned Value (EV) reports along the way. If problems do occur, project managers typically try to re-baseline the project with a new end date in order to again report acceptable SPI and EV metrics. This causes everything to look good again on paper, with no real benefit (and a delayed release date). There is a better way.

How does Scrum Avoid Those Problems?

Scrum delivers working software sooner and provides more accurate predictions of schedule, budget, and scope. Under Scrum, we’d plan the same ten-month project by dividing the work into ten, thirty-day sprints (see Figure 2). At the end of sprint #3 (30 percent into the project), we would have completed three iterations of development at a cost of $1.8 million. In the same time as a waterfall project and at the same cost, rather than a pile of specifications and approval forms, we would have produced three deliveries of demonstrated, shippable product. This is significant.

Figure 2

Figure 2. Ten-month (60,000-hour) Scrum project at 30 percent complete.

Figure 3 compares the two approaches (Scrum and waterfall) on the same fictional project. Notice that while both projects estimate the same number of hours to completion, only the Scrum project delivers working software every thirty days. Only the Scrum project monitors its schedule by the work delivered rather than through a sometimes misleading calculation. Finally, only on the Scrum project does the value accrue over time as each iteration’s software is delivered, rather than by a measure of how much of the planned work has been completed, regardless of whether any software is actually being delivered.

Figure 3

Figure 3. Waterfall “Schedule Performance Index” and “Earned Value” calculations vs. Scrum monitoring (using the same 10-Month (60,000-hour) project).
What Conclusions Can We Draw?

SPI is a poor barometer for measuring the health of a project schedule. “Earned Value” calculations create the false perception that actual “value” is being created during the early- to mid-phases of a project (when no software is produced). Scrum’s simple inspect and adapt mechanism tied to thirty-day (time-boxed) delivery cycles intuitively provides more reliable project status than traditional DOD metrics.

Article Rating

Current rating: 0 (0 ratings)

Comments

Anonymous, 2/13/2007 11:35:38 AM
Great article, where can we reference the figures mentioned in the article?
Anonymous, 2/27/2007 2:16:12 PM
The figures are now part of the article. Thank you for pointing out the missing information so that we could get it fixed.
Trevor Sterritt, CSM, 11/10/2007 4:51:07 PM
Great article, I agree with the conclusions at the end of article. Earned value and Agile practices have a very different opinion on what is 'value'. On Scrum projects, value means delivered software features. The value for projects using EV is completed tasks, whether or not those tasks deliver real world value to the users of the software.

However, EV does give you a way to compare schedule progress to budget progress. I don't think you should shy away from using it as a tool on Scrum projects where budget control is a consideration as long as you still use the product burn down as the main progress tracking tool.

John Hill, CSP,CSM,CSPO, 11/15/2007 2:19:47 PM
Trevor:

Since I wrote this article, several individuals at SolutionsIQ (esp. Tamara Sulaiman, PMP) have written a GREAT paper on using "Agile EVM" presented at Agile 2006 (link to this article is below):

http://www.solutionsiq.com/agile_index.html

Agile EVM circumvents all of the issues I surfaced in my article and I strongly recommend that organizations implement it when using Scrum (especially for PMI-oriented organizations). Agile EVM also provides the value you mention related to schedule progress and budget control in your comment above. Agile EVM additionally helps more process-oriented (waterfall-type) development organizations accept Scrum and other related Agile practices...

John H.
Trevor Sterritt, CSM, 11/24/2007 3:51:57 PM
John: Thank you for the reference to the Agile EVM paper. It was very insightful and provided details on how to adapt the EVM formulas for projects using story points estimating.
Anita Oppedal, CSM, 11/2/2009 7:24:09 AM
Hi
I am not able to find this link you are refering to:
http://www.solutionsiq.com/agile_index.html. This link is no longer value.
How to find this realted article? Do I need access to www.solutionssiq.com ?

You must Login or Signup to comment.