A Simple Way to Measure the Performance of Scrum Teams

6 May 2014


Measuring the performance of Scrum teams is the biggest challenge that most organizations face today. Teams become high performing only when they know their performance -- and continually improve on it.

I propose the following simple method to measure the performance of Scrum teams; you can use this if it works for you.

Parameters
  • User stories planned versus user stories delivered -- Measures planning effectiveness, efficiency
  • User stories delivered versus user stories accepted -- Measures quality of deliverables
  • Defect-removal efficiency (DRE) -- Measures defects-removal rate
Note: If the team is working on production defects, then production defects can be replaced with the user stories.

Example



Note: You can define your own GAR Scale based on your requirements.

Advantages of this method
  • Very simple to apply and easy to understand
  • Can be applied at the release as well as sprint levels
  • Required data can be obtained easily from Rally or any other tool
  • We can compare multiple heterogeneous teams
  • Helps in identifying weak areas, so that necessary corrective/preventive actions can be taken in a timely manner



Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.



Article Rating

Current rating: 1.3 (4 ratings)

Comments

Carlos Monroy Nieblas, CSM, 5/6/2014 7:00:28 AM
The Agile manifesto proposed to give priority to individuals and interactions over processes and tools; this measuring procedure would facilitate an environment where everybody is "chasing the number".

Also, in the example, it would too late for the team 2 to wait until the end of the sprint to find out that it was 67% of the user stories and try to find out the root cause (that is one of the benefits to have daily stand up meetings, as well as one of the primarily objectives of the ScrumMaster)
Tim Baffa, CSM, 5/6/2014 4:24:30 PM
Marmamula, any performance evaluation of a scrum team may involve a metric based on user stories accepted and user stories delivered. However, one area that a scrum team should never be evaluated on is your first bullet point - "User stories planned versus user stories delivered".

While effective communication between the team and the PO / stakeholders is encouraged in Scrum (refinement, manage expectations), the team has very little if any input or influence over user stories "planned". This concept seems to be a carry-over from the waterfall days, where team performance is evaluated by comparing their production against the long-term goals of the organization.

This needs to be strongly discouraged, as the team is really only in control of what they deliver every sprint against what they accept in each Sprint Planning Part 2 meeting as either a subset or entirety of the offer made in Sprint Planning Part 1.

This is critical to effective Scrum and team growth. Teams should never be judged by how they deliver according to organizational goals. It is up to the Product Owner and their prioritization practices to ensure that the team is working on the highest-priority items with the greatest business value.

Also, your DRE metric would likely go down the more mature a scrum team becomes with test-driven development and continuous integration. Teams should not be penalized in the event that they are not uncovering defects within a sprint, especially if their coding and technical practices are improving to the point that defects become a rarity.
Charles Cain, CSP,CSM,CSPO, 5/7/2014 12:26:06 AM
Comparing teams is a lot like comparing apples to walnuts, they just shouldn't be done. To reduce the output of teams to nothing but metrics in a spreadsheet is setting the team up for failure and depression.

Team #1 in the example above could be working on display code. Team #2 could be working on the database that will feed the display code. Now let's say that Team #1 is using the same code platform to build the display, but Team #2 is using a new database system that they have no experience on.

The first couple of sprints Team #2 will have a lower DRE score and Team #1 will have a higher DRE score because of their knowledge already in place. If someone was to look at the above spreadsheet, they would think that the members of Team #2 had problems and would then perhaps try to send in more resources to help which is one of the worst things that could be done.

As Tim stated, teams should not be judged by delivering anything but the highest priority items as determined by the team and the Product Owner. And you should never compare teams because of the differences between them.
Michael Kuesters, CSP,CSM,CSPO, 5/7/2014 8:23:12 AM
Edward Deming formulated nearly 70 years ago the 14 Principles of TQM.

Two of these principles are:
- Eliminate numerical quotas for the workforce and numerical goals for management.
- Remove barriers that rob people of pride of workmanship, and eliminate the annual rating or merit system.

Numerical goals always have the problem of creating so-called "perverse incentives".
For instance, Team X is highly ambitious and always assumes high risk stories, to generate learnings and value for the company.
Team Y is full of weasels who have found a way to water down stories until there is no business value left, then excel at getting them "Done".

Now, Team X will fail delivering stories left and right. What is the resolution to the root cause? Stop Team X from engaging risks? Stop them from learning stuff? From producing value, facing challenges and having fun?

The root cause of not getting stories "Done" is, at least in the teams which I run, that my teams are engaged in unpredictable, challenging, creative work when innovating new stuff.


Unfortunately, your metric contradicts one of the core Agile tenants, producing an environment where failure is not only accepted - but encouraged.

"Fail fast, move on".
So much better than "Stay green until inevitable doom".
Michael Kuesters, CSP,CSM,CSPO, 5/7/2014 8:32:36 AM
Ha, Tim, you caught a good one:
"Also, your DRE metric would likely go down the more mature a scrum team becomes with test-driven development and continuous integration. "

Yes!
The DRE metric really penalizes teams that only have rare-and-inbetween but tough defects.
It rewards teams for producing tons of defects and fixing them quickly "Hey look, Team A fixed 780/1000 defects in 2 weeks, while Team B got only 1/2 defects fixed" ... now: which team is better?

Again, a perverse incentive: the bane of every Metrical Manager.
Zach Bonaker, CSP,CSM,CSPO, 5/7/2014 10:04:34 AM
Somewhere along this journey, I was under the impression that - when it comes to working software - it's either done or not.
Michael Kuesters, CSP,CSM,CSPO, 5/8/2014 6:43:26 AM
Zach, yes of course "it's done or not".
But there's still that thing called "Production defects", sometimes also labelled "problem" in ITIL.
A good SCRUM team would strive to produce "Zero Defects in Production" - but they just happen in complex architectures.

It should be any team's intrinsic motivation to make sure that a "Done" story never pops up again.

Therefore, introducing a metric which rewards teams for producing highly fragile software is definitely a bad idea.

Then again, the only metric that *truly* matters "Does the team build software that makes the customer happy?"
Marmamula Prashanth Kumar, CSM, 5/13/2014 1:42:51 AM
Thank you very much for your valuable comments...
I want to clear few things here.
Here I am not comparing Planned of one team with delivered of one team.
This can be applied to any team if the work complexity is higher let them plan for lesser user stories. We are only comparing the ratios of final values.
However this is just my idea, if it works you can use it.This idea is not against any of the Agile principles.
Marjorie Blanco, CSM, 5/19/2014 10:54:09 PM
Interesting article. I agree with not comparing agile team. This method only tells you how the team performed for a give sprint, how you consider expanding by comparing against previous sprints (time-series graph)?
I would also add that this is tool that the team can use during retrospective.

You must Login or Signup to comment.