Agile Project Reporting and Metrics

29 July 2013


Agile processes are generative, not prescriptive. Processes need to evolve as needed, not be prescribed up front. A prescriptive approach generates complex and complicated processes, whereas a generative approach begins with a set of simple processes and adds others as they are needed.
-- Jim Highsmith, author of Agile Project Management
 
I have written this article to address an issue I've recently run across in my work: The PMO is finding it difficult to adjust to the Agile process, and this is affecting the reporting of the R&D team. Management is in the process of moving the ALM (application lifecycle management) process into more of traditional process to align with the PMO. I would like to suggest an alternative approach.

To give some background: Agile recognizes that most effective software processes cannot be defined up front; it is a continuous process. Traditional ALM (application lifecycle management) processes focus on defining and enforcing processes and spend very little time identifying and delivering what customers need. Although their aim is to improve consistency and quality, this often comes at the expense of productivity and delivery. Traditional ALM has also uses monolithic, heavyweight tools, causing fragile builds and ineffective hand-offs between development, testing, deployment, and operations.

Agile ALM frees companies from heavyweight processes and tools. Using continuous builds, the customer regularly receives working software that allows the customer to gauge the product and provide corrective measures before it is finalized. While traditional ALM applies tools to enforce predefined, standardized process, Agile ALM is the application of tools to support people and the processes that best suit the way they actually work when creating great software. 

A typical product development company has an issue management process (ticket handling for bugs/defects), change/feature management processes, and new customer implementation processes. The PMO is a centralized resource for all the IT project teams, monitoring project delivery, with a wide range of projects with defined goals and time frames. The PMO is responsible for resource planning, including staff competencies, assignment, and capacity planning; metrics and reporting to management; and definition and standardization project implementation and delivery processes, guidelines, training, and other project support services. Project managers need to consistently communicate about and report on their projects across business units and development methodologies to the PMO, so the PMO can present a single dashboard across a breadth of projects within a COO's control. It is not surprising that the PMO is often heavy-handed, imposing a strict approach to metrics and guidance across all projects.

Among the above responsibilities, the greatest impacts on the Agile project team are:
  1. Providing the dashboard to all stakeholders
  2. Providing metrics for processes, project deliveries, etc.
  3. Defining standard development practices that will support and can be easily mapped to traditional project reporting and practice
Agile processes must exist within the PMO infrastructure. This requires that the Agile team be willing to conform (at least externally) to traditional project management practices. Not only must the Agile project adapt but the PMO must as well. Here there's a benefit from the ability of Agile teams to accurately measure the status of a project and adapt to changing requirements.

Legacy PMO tracks projects on the basis of project schedule, budget, issues, and risk. This data is largely fed through project management tools (MSProject, etc). The PMO tracks time and cost estimates from project task dependencies across the phase of the project life cycle. Agile teams must communicate effectively, by using language and metrics that the PMO can appreciate. Agile teams report on velocity (the rate at which the team delivers a story), burn-down charts (the trend of remaining effort), test coverage, and running test features. These do not effectively help reporting on time, budget, and risk analysis. Hence the Agile team needs to identify and bridge this reporting gap.

Sample Agile reporting

Release in Agile ALM mapped in the project plan:

Saji-SampleAgileMetrics-IMAGE-1-(2).jpg
 
Release backlog:

Saji-ReleaseBacklog-IMAGE-2-(2).jpg
 
Traditional project plan and mapping user stories to the project plan::

Saji-MppPictures-IMAGE-3-(2).jpg
Metrics such as earned value analysis (EVA) and cost to complete are certainly valuable to all stakeholders.

Key earned value management terms defined

(Note: The term value in EVM does not refer to business value. The term refers to value as expressed by the project or program budget.)

Planned value (PV): The value of the work planned to be accomplished based on the budget (in dollars or hours)

Earned Value (EV): The integrated value of work actually accomplished based on the budget (in dollars or hours).

Actual Cost (AC): The cost incurred for that increment of work.

Budget at Complete (BAC): The budget assigned to complete the work.

Estimate to Complete (ETC): The forecast amount to complete the remaining work (in dollars or hours), based on past performance.

Estimate at Complete (EAC): The forecast total amount for all work in the project plan, based on past performance.

 
Metrics
Formula Metric Analysis
Planned Value BAC * Planned
Percent Complete
The planned value indicates how much value was planned to have been generated by a particular milestone or point in time.
Earned Value BAC * Actual
Percent Complete
 
 
The earned value indicates how much value has actually been generated at a particular milestone or point in time.
 
 
Cost Performance
Index (CPI)
EV/AC This metric indicates how many cents have been "earned" out of every dollar spent. It measures cost efficiencies.
 
 
Schedule
Performance Index
(SPI)
PV/AC This metric measures schedule efficiency. It indicates how fast you are
progressing against the rate of progress planned.
ETC (BAC-EV)/CPI  
 
This metric is the forecast amount to complete the remaining work.
EAC  
 
BAC/CPI OR
AC+ETC
Forecast cost for the total planned work.
 
Agile story points can be used as the measure of work planned and work performed, providing the basis for all of the traditional earned value calculations. Percent complete can be derived by dividing the number of complete sprints by the total number of planned sprints. Actual percent complete can be derived by dividing the number of story points completed by the total number of story points planned for a release. To calculate the initial baseline in Agile, five data points are required:  
  1. The number of planned sprints in the release backlog
  2. The length of each sprint in calendar days
  3. The number of story points planned for the release backlog
  4. The budget planned for the release backlog
  5. The start date of the project
 
To calculate EVM in Agile:
  1. Number of sprints completed: This measures the expected percent complete when divided by the total number of sprints planned.
  2. Net story points added (this can be a negative number if story points are removed from the release.) This reflects the net change in work planned for the release, in effect re-baselining the release. Actual cost to date at each boundary: We are measuring results to date, and therefore we use the cumulative actual cost as of that sprint boundary.
  3. Cumulative story points completed, which measures the total amount of work completed for the release as of that sprint boundary.
 
Agile dashboard for traditional PMO
 
Item
  Definition
Budget at Complete (BAC)   The planned amount you expect to spend
Actual Cost (AC)    
The actual cost to complete the work
 
 
PRSP
  Planned release story points for the release; story points are defined at the product backlog level
Expected Percent Complete (EPC)   Current sprint(n) / Total planned sprints
Actual Percent Complete (APC)    
Story points completed / Total planned story points
Planned Value (PV)   PV = BAC * EPC
Earned Value (EV)   EV = BAC * APC
Cost Performance Index (CPI)   CPI = EV / AC
Schedule Performance Index   CSPI = EV / PV

 
Saji-AgileDashBoardPMO-IMAGE-5.png

Team dashboard using a task board
In Agile, it is a common practice to maintain the task board on a wall to help everyone visualize the project status. It's a visual process management system that tells what to produce, when to produce it, and how to produce it.

The project status as a dashboard

Saji-TaskBoards-IMAGE-6-(2).jpg
 
Defect metrics as per release backlog

Saji-TaskBoard2-IMAGE-7-(2).jpg
 Saji-DefectIteration-IMAGE-8-(2).jpg
 
 
 
 

Article Rating

Current rating: 4.3 (9 ratings)

Comments

Glen Wang, CSM, 7/29/2013 9:46:36 PM
So agile still need a project manager to manage the report and metrics as it's complex:)
Greg Manto, CSP,CSM,CSPO, 7/30/2013 6:28:28 AM
Saji, kudos on providing an approach to bridge this "gap". I am curious who your intended audience is for this article? I'm assuming that this is intended to give Agilists a richer vocabulary / toolset for being effective when collaborating with their PMO's. If that is accurate, have you also considered creating a similar article for giving the PMO's a richer vocabulary / toolset toolset for being more effective when collaborating with their Agilists? If so, it would be interesting to hear how that went.
Kevin Pedersen, CSM,CSPO, 8/1/2013 2:38:13 AM
This is a great article. I have been struggling to understand traditional PMO reporting techniques and this gives me a good idea of how I can bridge my agile understanding across to those methods. Thanks very much!
Dave Spooner, CSM, 8/5/2013 8:22:28 AM
Very useful article. Glen, I am not sure if that was tongue in cheek, but if not I think you have misinterpreted the article slightly. Agile can be reported on and progress tracked very easily, and there are many resources outlining how to do this. This article is very useful if you are forced to shoe-horn agile into traditional reporting structures, which is often the case in a larger or more traditional organisation. I know I face this challenge myself, running a Scrum development team within an organisation that is not a software house. Thanks Saji.
Sameer Chudaman Patil, CSP,CSM,CSPO, 8/21/2013 1:09:32 AM
First I would like to say agile team should not map or report to PMO and if they do so it’s not mandate that they should report/communicate in language of PMO. This is what exactly talked in detail when you talk about transition. It’s about mindset which we need to change and adapt new ways of working together.
I have seen burn up in your case as specialty base Second, release burn up never been reported as specialty base, it always reported in terms team efforts under which functional expertise are spread across.
For estimating and tracking your project if you know velocity and story point which has been estimated by one team or by many teams (if more than one scrum team involve) then you can easily find imp parameters in Project management world. So if you do so then I don’t think even PMO/PM should face problem in tracking, managing, budgeting project .
As mention in your defect matrix: It’s advisable to not to maintain defect matrix dashboard unless you have special sprint to fix defects.
Also looking at your article it looks like that your agile team does not include product owner role who eventually take to communicate to all stakeholders on tracking and progress of project. Do you have PO in you AGILE ALM?
So looking at data provided you are doing scrum but not a “Scrum”.
Rohit Ratan Mani, CSM, 8/21/2013 6:50:02 AM
Great article Saji.
I have also faced similar queries from PMO team whenever I am getting the projects reviewed.
A similar traditional approach for review is followed by Quality teams. It would be insightful to capture how Quality policies should modify for Agile Projects from traditional approaches.
Balaji C R, CSM, 11/20/2013 4:19:32 AM
Very Nice article, gives reporting information for the PMO and align to the overall dashboard expectations. Good one.

You must Login or Signup to comment.