Get certified - Transform your world of work today

Close

SPOC: Story POint Cost

How to measure and track the cost of your Agile projects

18 March 2015

Mahfoud AMIOUR
SOFTENIA Solutions

Introduction

Metrics and measurement are important in software development projects. They're needed for planning, budgeting, organizing, and controlling the work. They're also necessary for continuous improvement.

Scrum provides three main metrics: velocity, sprint burn-down, and release burn-up. These metrics focus mainly on the development team and on the product. They don't directly address the cost of the project.

In this article I present SPOC (Story POint Cost) as a metric to measure and track the cost of user stories and consequently the cost of Agile projects. I've developed this metric during my work as an Agile/Scrum consultant with different clients (finance, industry, services). The metric has been used in several projects during the last three years. The main objectives of the SPOC metric are the following:
  • Providing Scrum teams (and managers) with a simple means for measuring and tracking the development cost all along the project.
  • Having a concrete measure of story point cost will reinforce team awareness about the budget. This allows the team to register cost reduction among its objectives for continuous improvement.
  • Keeping a history of the costs will allow for more accurate budget estimates for future projects (estimation based on accumulated experience of the teams).


SPOC (Story POint Cost) metric

SPOC stands for Story POint Cost. It represents the number of man days (M/D) spent to realize one story point. The SPOC metric can be measured at three different levels:
  • SPOC at the sprint level: measured at the end of each sprint
  • Intermediate SPOC: the average cost after n sprints.
  • SPOC at release level: measured at the end of the release (last sprint)
Measuring the cost at different levels allows regular inspections at different moments in the project's life cycle and helps identify corrective actions when needed.

1. Sprint SPOC

Sprint SPOC is measured at the end of the sprint as follows:


The figure below shows an example of sprint SPOC for the first sprints of one of my projects. The project was about implementing a financial application.

2. Intermediate SPOC

Intermediate SPOC represents the average cost of one story point after n sprints. It's the result of the division of the accumulated man days consumed during the different sprints, by the total size realized (accepted stories).



The table below shows an example of intermediate SPOCs for the financial application project.


3. Release SPOC

Release SPOC is measured at the end of the release (i.e., the last sprint). It's the result of the division of accumulated man days consumed during the release, by the total size accepted.



In our example, the release SPOC is 6,25 M/D.

Graphical representation of the SPOC metric

This figure is a graphical representation of SPOC evolution at the three levels (sprint, intermediate, release).


Case study

This section presents a case study that illustrates the usage of the SPOC metric with one of my clients.

The context

  • The client of the case study is a major player in the financial industry located in Paris, France.
  • The metric has been used on three sequential projects between 2013 and 2014 (financial applications for internal and external use).
  • There were several Scrum teams involved (between two and five teams of eight persons each, depending on the project).

SPOC history

The graphic below shows the history of SPOC at release level:


The SPOC of Project 1 was around 7,57 M/D. This was the first project managed using Scrum by the client (only a small pilot project had been done before). The duration was 10 sprints of 3 weeks. Two teams of 8 persons each worked on the project.

Analysis of SPOC after project 1

This SPOC of Project 1 was too high. A retrospective involving the participating teams has identified the following reasons:

  • The teams were not mature enough (were new to Scrum).
  • Some stories were underestimated.
  • Most of the work done during the first sprints was dedicated to the implementation of the whole architecture of the system. There were no functional features with added value to the client. This was an error. Due to some changes in the scope of the release, much architectural rework was necessary. This has considerably reduced the velocity of the teams.
  • There was some coordination overhead due to the presence of two teams working on the project. The repartition of work between the two teams was not optimal (there were many dependencies) between the teams.

Corrective actions

The challenge was to improve the SPOC in the next projects. Based on the lessons learned, some corrective actions have been identified:
  • Minimize dependencies between the teams. This has been achieved by a better split and repartition of user stories among the teams.
  • Better coordination between the teams: regular Scrum of Scrum meetings; setup of common working group coping with shared aspects of the project (architecture, data model).
  • Better definition of sprint content. Functional stories with value for the client have been assigned to the teams since the first sprints. This helped the teams to do technical architectural work all along the release (a work has been done before starting to sprints on order to define the bases of the architecture).
The corrective actions above were a real success for the teams. The SPOC went down to 4,22 M/D in project 2. In project 3, the SPOC has been reduced to 3,66 M/D (the teams have not been changed).

Conclusion

In this article we proposed "SPOC" as a metric to measure the cost of story points in Agile/Scrum projects. The metric can have different uses:
  • Keeping track of story point costs, which allows the teams measuring their performances and identify improvement actions
  • Estimating the budget of future projects based on the experience of the teams
Any comments are welcome.


Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.



Article Rating

Current rating: 1.9 (7 ratings)

Comments

Deepak Joshi, CSP,CSM,CSPO, 3/22/2015 7:47:10 AM
Hi Mahfoud, good one and nicely explained.
Mahfoud AMIOUR, CSP,CSM,CSPO, 3/23/2015 3:25:00 AM
Hi Deepak, Many thanks. I am glad you appreciate the article. Kind Regards.
JOHN JAMES, CSM, 3/24/2015 11:40:53 AM
Hi, Very Good Article and very informative Many thanks for this
John
Bethan Timmins, CSP,CSM, 3/25/2015 10:15:28 AM
Hi Mahfoud, a good article that I found really interesting.

I previously have done something similar but at a higher level. I have a sprint cost for the team, and used the sprint cost to map against t-shirt estimates of epics. (then internally kept a record of the number of points really completed to meet that epic) I tended to keep away from doing this at a lower level as I found that as the velocity of the team changed slightly from sprint to sprint so did the cost of the story points.

I have always liked the idea that the "value" of a point was arbitrary to business stakeholders and it was used more for the team to commit to and estimate complexity. my worry was that all of a sudden putting a "value" to it the level of scrutiny on the team would increase.

Have you encountered any issues with doing this at a lower level. If so what issues did you face and how did you overcome them?

Bethan
Krishnaswamy Babu, CSM, 4/17/2015 2:24:41 PM
Good article. Story points are a measure of the implementation complexity and may not have a linear relationship with person-days. However, for a metric such as SPoC to be universally useful, I imagine more experience in its measurement might be required.
Mike Dwyer, CST,CSP,CSM,CSPO,REP, 4/17/2015 2:30:46 PM
Help me understand the efficacy of comparing different teams story points against the number of man days consumed.
i would be very interested in seeing 3 teams size the same story, then deliver it. I

Team 1 sizes it a 3
team 2 at 5
team 3 at 8
If team 1 completed the story twice as fast as team two and 3 times as fast as team 3, then we could extrapolate the relationship of man days to story points from is a reasonable thought.
However if team 1 take twice as long as team 2 and team 3 does it in half the time as team 1, this might suggest that story points are unitless and reflect the perception of a team. We also might want to study measures a bit more.
Alan Dayley, CSP,CSM,CSPO, 4/17/2015 3:21:15 PM
Story Point Cost appears to be a way of connecting Story Point Velocity to cost. The negative effects of trying to "increase velocity" are well documented (http://agileotter.blogspot.com/2012/09/14-weird-observations-about-agile-team.html). It seems to me that the negative effects of trying to "decrease story point cost" would be similar, since the metric is closely related to velocity.


How did you avoid any negative behavioral and product effects such as would be expected when attempting to improve Velocity?

What effects on the team behavior did you see when teams were aware of this measurement?

When Story Point Cost was going down, what was happening with other measures of quality and value?

Companies are very adept at measuring costs. At a less granular level, the company already knows how much money a given team burns in every Sprint. How does higher granularity of measuring cost improve the product or delivery of value?

Story points are not a measure of value. Wouldn't determining an appropriate metric of value delivered per Sprint be more important than tracking cost in yet another way?
Gene Gendel, CEC,CSP,CSM,CSPO, 4/17/2015 4:11:14 PM
I guess I am a bit puzzled with the math of this... dividing Hours by Story points is effectively dividing time scale by time scale, if we agree that story points are “hours” in team’s language (universal language). Mike Cohn actually describes what a story point is in his https://www.frontrowagile.com/courses/agile-estimating-and-planning/overview really well. There is the whole section on comparing story points and ideal time/days.
I guess, if I saw a division of Story Points by Hours then I would be less puzzled. I would then think that you are trying to calculate how many hours a single story point is worth. But Hours divided by story points…I am just puzzled a bit.
Also, I am failing to see a connection between hours and dollars. This should be actually a very basic math, if you know pay rate of each team member.
I am not suggesting that organizations should note be attempting to materialize Story Points, I am just struggling with logic behind the calculations. I think Rally Tool is attempting something similar out of the box. But again, they have a “dollar” concept used as well. Here is some reference:
https://help.rallydev.com/agile-evm
Ron Jeffries, CST,CSD,CSM,CSPO,REP, 4/17/2015 6:32:04 PM
Two thoughts:

First, this metric can readily be gamed. If an organization were using it for other than info inside the team, I would in fact game it. So would any rational employee. As for use inside the team, simple burn charts or story task boards should work as well. I can see that a big company might want this: I do not consider it to be particularly in the spirit of Scrum and Agile.

Second, value trumps cost. A much more useful metric would show the ratio between value and cost, or simply the flow of value.

Summing up: interesting idea, overly complex, easy to game, and not ideally focused.
Joseph Pontes, CSM, 4/19/2015 7:41:52 AM
I can appreciate what this is trying to do but there are 3 points I'd like to make:

The first is a fundamental problem with trying to associate story points with man hours to begin with. Story points by their nature are meant to abstract away the hours it takes to do something and focus purely on size of effort. Something "medium" could take an associate 60 hours but a principal 12 hours... and yet it's still just a "medium" in story points. Trying to establish a connection between hours and story points is what non-Agile business people struggle not to do, and we should not contribute to that struggle by making the connection ourselves.

The second is that if you really want to come up with some cost of story points despite my first point above, then it should factor in salary. An associate taking 60 hours to do something might actually cost less than a principal taking 12 hours, or it might cost more - bottom line is that by not recognizing that not all man hours are made equal then any attempt to associate man hours to Story points will be deeply flawed (as well as misguided).

The third echoes some of what others have said above - that this measurement by its nature encourages the wrong behaviors. Teams can improve their cost per story point very easily by just gradually implementing story point inflation. What was a 2 one sprint becomes a 3 the next and a 5 the sprint after that. Suddenly and magically it will appear as if more story points are being done per man hour, yet noting will have really changed other than an undermining of the real use of story points.
Mahfoud AMIOUR, CSP,CSM,CSPO, 4/19/2015 8:33:34 AM
Many thanks to every body for your comments.
I would like to give some additional precisions.
The measurement proposed is for Agile teams to allow them (continuously) improve their performances (including cost reduction). It's not intended to control them. Of course, this has to be explained to the teams (and to the whole organisation if the measurement is used outside the teams). Team members need to be reassured that the measurement is intended for continuous improvement, not for blaming them in case of project failure.
I would like to note that the SPOC metric is just a measurement. It’s not a linear correspondency between Story Points and Man/Days. In the case study, estimation of stories was based on story points. The estimation of taks was based on hours.

I would also like to mention that the purpose of SPOC is not to compare the teams. The projects indicated in the article are multi-teams projects (up to 5 teams working together on each project). Story estimations is done commonly by the different teams (representatives from each team). This allowed to have common shared sizes of the stories. So the measurement concerns the whole project and not individual teams.

To avoid the temptation of gaming and inflating Story Points, it's very important to explain the objective of the measurement to the whole organisation (teams, management, stakeholders …). In empirical approches (as Agile), measurement is done for improvement and not for blaming. Having trust in the teams is also very important. Otherwise the measurement will not work.

More experiments are needed to see how the metric works in different contexts.

Kind Regards
Mark Levison, CST,CSP,CSM,CSPO,REP, 4/19/2015 3:38:56 PM
Mahfoud - I've only a couple of minutes on the weekend. I understand the appeal of what you're trying to do.

Some cautions - the whole point of story points is to move us away from the focus on hours.
Your approach assumes that story points remain constant and that estimation is good. Anyone who has attended one of my CSM classes should be able to cite at least 5 reasons that estimations are bad. You also mention a desire to improve estimates. This one quote from Kahneman & Tversky, 1977:

“However, the evidence suggests that people are insufficiently sensitive to distributional data even when such data are available. Indeed, recent research suggests that people rely primarily on singular information, even when it is scanty and unreliable, and give insufficient weight to distributional information”

Summarizes the problem.
----

For more thinking in this line see: http://www.infoq.com/news/2011/11/velocity-highsmith and http://agilepainrelief.com/notesfromatooluser/2010/02/misuse-of-velocity-in-agile-projects.html

Instead of focusing on measuring cost why don't you focus on value. If you focus on cost you will get lower costs but probably miss value.
Dave Prior, CST,CSP,CSM,CSPO,REP, 4/19/2015 5:54:34 PM
Hi Mahfoud,

I understand the appeal of what you are aiming at. But I am concerned about how it plays out in the case study. My fear with this is that someone would look at a teams SPOC and use it to judge them and say, "too high" or "too low", or "why can't your SPOC be more like Team Awesome over there." It seems like this is exactly what happens in your case study. Regarding the analysis of SPOC after Project 1, you say that the "SPOC of Project 1 was too high". I do get that you want for this to be something that will fuel the desire the team has for continuous improvement, but what I'm more worried about is that with publishing something like this internally, you'd get the "Project 1 was too high" from management and it would be a judgment placed on those working on Project 1. Or, they'd be slammed because the SPOC of the team working on Project 2 was "better".

From what you describe in your case study, if the SPOC of Project 1 was 7.57, wouldn't that simply mean that the SPOC was 7.57, and not that it was too high or too low? It is what it is. If the team can find a way to change it, that is obviously a good thing, but I think one of the challenges with metrics is that they are a bit like knives. They are useful tools, but they are also sharp, pointy and can be used for harm as well as good.
Fran Paterson, CSM, 4/20/2015 3:36:19 AM
Reminds me of function points - look at the number of outputs/screens, the data (sources, manipulation and storage), factor complexity etc and map against the effort days expended for the base figure, then look at team maturity as a team, technology used and come up with a weighted value to use when estimating the next and so on.
Mahfoud AMIOUR, CSP,CSM,CSPO, 4/20/2015 3:53:23 AM
@Mark: here are some precisions:

SPOC doesn't assume that estimations are constent and good. The measurement is done at the end of each sprint and only "Done" stories are included in the measurement. Teams can refine/improve the estimations whenever they want. The SPOC is updated accordingly.

SPOC doesn't change the way we estimate. The teams keep using story points to estimate stories and hours to estimate the tasks implementing them.

Measuring costs doesn't mean to "forget" the other important metrics such as business value and quality. I systematically encourage/help teams in seting up metrics for Value/ROI and Quality (I hope to be able to publish some experiments about that).
Mahfoud AMIOUR, CSP,CSM,CSPO, 4/20/2015 4:06:26 AM
@Dave: You are right. We need to be careful when measuring costs. It's important to avoid judging and blaming the teams. An Agile coach needs to explain and educate the whole organisation that measurement is used for improvement (not for judging and blaming).

The use of "Too high" and "Too low" qualifiers is done only in the article. They are not used as part of the reporting.
Mark Levison, CST,CSP,CSM,CSPO,REP, 4/20/2015 10:16:25 AM
Mahfoud - I will close my thoughts with two points.
- I suspect most of us who deal with estimation real don't understand what is going on. Both psychology (and why we're so bad at it) and statistics I think you're spotting significance in noise (see: http://kadavy.net/blog/posts/aa-testing/ for a good example of this). I remind all teams I coach be careful what you measure you will get it. Low cost story points aren't related to anything useful.
- Scrum is about simplicity and this idea feels like the opposite. To understand the importance of simplicity you might want to read - Andrew Haldane's speech - "The Dog and the Frisbee": http://www.bis.org/review/r120905a.pdf

Cheers
Mark
Mahfoud AMIOUR, CSP,CSM,CSPO, 4/20/2015 11:32:00 AM
Mark, thanks for sharing the links.
I understand your thoughts but I don't agree with them.
In my humble opinion, measuring in order to decrease the costs is usefull. In the case study I presented, there was real needs from the teams and the organisation to measure the cost.
I don't see how this measurement is making things complex. Of course we need to be carefull how we do it. In average it took me about 15 minutes each sprint to gather information from the different teams (about 5 teams).

Best Regards

Mahfoud
Mark Levison, CST,CSP,CSM,CSPO,REP, 4/20/2015 12:37:45 PM
Mahfoud - I don't doubt how easy the metrics are to collect. I think that they fundamentally misunderstand Scrum.

In many organizations we need to overturn the tyranny of cost over value and madness of over measurement. You appear to be giving succour and support to bad habits.
Mahfoud AMIOUR, CSP,CSM,CSPO, 4/22/2015 6:27:13 AM
I don't think that measuring costs is giving a succour to bad habits. Of course we need to be carefull how we to make the measurement.
Among the lessons I learn't from the experiences reported in this article is that measuring costs can be compatible with Agile. As the measurement has been defined and done with the teams, those teams have gained in maturity. Besides of attempting to maximize the value provided to the clients, the teams became aware about cost and budgeting aspects.

Kind Regards
Mahfoud
Fabio Leme de Almeida, CSP,CSM,CSPO, 5/4/2015 2:36:31 PM
Mahfoud, I like and I think an interesting idea. Very good article.
Mahfoud AMIOUR, CSP,CSM,CSPO, 5/9/2015 3:54:58 AM
Thanks Fabio. I am glad you appreciate the article. Kind Regards

You must Login or Signup to comment.

 
The community welcomes feedback that is constructive and supportive, in the spirit of better understanding and implementation of Scrum.


 

Newsletter Sign-Up

Subscribe