Agile Performance Appraisals

14 March 2012

Arpit Gautam
Kronos

We all do the crazy exercise of rating team members every year. And, looking back at a software development industry that's almost 50 years old, we know certain things for sure: Software is built by teams, not individuals. Moreoever, each individual needs to actively collaborate to produce quality software. This means that everyone on the team needs to take collective ownership and help each other, because the motive is not to be a hero but to build an end product of the utmost quality and predictability.

Thanks to Scrum, we do this every day and produce good software. But also we incubate the evil of rating team members, which is anti-Agile (see "Tracking Individual Performances in Scrum"). We know that it is insane to compare work of two individuals and two teams — but we do it. Individual performance assessments are a reality. Yet Scrum supports the doctrine of cross-functional, self-managed teams in which individuals help each other in completing the tasks to deliver value. The ownership of completing user stories lies not with any one individual but with the complete, collective team.

This contradicts the appraisal process, because it is almost always centered on how an individual is performing. Inevitably, a team using Scrum has to quantify and produce matrices around an individual's performance. This raises a handful of questions:

  • Can we contribute the success of a sprint (and/or user stories) to any one of individual, when the team has completed it?
  • How can we quantify and take into account the fact that most developers sometimes help each other complete a certain task or a user story?
  • How do we encourage team members to help each other selflessly?
  • To summarize, can we measure individual velocity?

There are few ways one can address this:

Use velocity to measure individual performance. Can we find the velocity of a developer that will tell us how much he or she has achieved in a year? As we all know, velocity is a measure of how much a team can achieve in a particular sprint. It can vary across teams. Finding it for every individual in the team is difficult, if not impossible, as user stories are completed by teams and not individuals. So velocity is a team thing; we should leave it at that.

Produce a task matrix for every individual. Can we track tasks completed by an individual and find out what percentage of a user story is completed by him or her? If we knew that, it would be easy to know who has produced the most output within a team. Unfortunately (or fortunately!), individuals do help each other in tasks. So just because someone is completing more tasks doesn't mean he or she is "better" than the others at the job.

Give ratings to teams instead of individuals. What if we rate teams and appraise them instead of individuals? Now the problem boils down to how to compare two teams. Velocity and acceleration are, again, not our friends here. Comparing teams is only possible if we can measure equivalent user stories and find out how the teams are handling those, which is difficult given the divergent nature of projects in an organization.

Handle it with Agile. This is by far the most practical way of assessing an individual. Let's jot down an overview of how we'd handle the process in Agile way:

  1. Determine goals for every individual. These goals need to center on the individual — for example, how can he or she achieve these goals and become better at software creation? When we set each goal, we need to identify a nonambiguous acceptance test for it, and a way to demonstrate whether or not it is met. Let's take, for example, a goal of automating acceptance tests for the product. If this is 60 percent complete, we will say the goal is met; if it's 60 to 80 percent complete, we can say the person has exceeded expectations; if it is 80 to 100 percent complete, then the person has far exceeded expectations. The thing to notice here is that it looks like an individual goal at first glance, but it does require collaboration, so either everyone will meet it or no one will. Therefore, it's a good contender as a goal for every developer on the team.
  2. Create an appraisal backlog for the individual, with goals and acceptance criteria for each goal. Capture the information about how and why the tests for these goals are failing at this point in time.
  3. The product owner for this backlog should be the mentor for the individual. He or she will prioritize the goals at this stage.
  4. Now the mentor needs to assign story points to various goals. For example, the acceptance test goal discussed above can be assigned a value of 8 on a scale of 11 points, as it requires quite a bit of effort.
  5. The mentor is free to add more goals to an individual's goal backlog after due discussion with him or her.
  6. The mentor and individual need to identify a sprint length, after which they will review the product backlog and have a demo. Most organizations already do this exercise and have one-on-one meetings once every month or two. The different thing we need to do is to qualify the goal on the nonambiguous acceptance criteria and produce a rating for it. So, for example, after a month the acceptance test automation is reviewed and it's discovered that only 50 percent automation has been achieved up to now. The mentor can then talk to individual(s) to determine whether the goal is too difficult, whether they are facing any blocks in fulfilling it, or whether an individual is not just able to meet the goal. The mentor can then play the role of enabler and help the team member(s) achieve the goal — or revise the goal, decreasing its weight and acceptance criteria to make it more achievable. This makes the goal easier but less rewarding.
  7. Next, the sprint's backlog will be derived on outcome of this sprint. If individual has failed to meet a goal with 5 story points, he won't be given a goal of 8 story points in next sprint. He should be given a goal with 5 or fewer points. This will ultimately help his or her overall growth.
  8. In the demo, make the acceptance tests pass and qualify the goal as done. Additionally, you could have optional criteria for exceeding or far exceeding the goal (note that we determined a percentage range of automation when we discussed the goal in the first step).
  9. In the ultimate performance review, one could look at the matrix of goals and qualified acceptance criteria. This matrix should capture the information from every iteration/month.
  10. As we saw in earlier approaches, it is extremely difficult to compare one project to other. In the same way, it is not practical to compare two individuals on the basis of features they helped add to a product. This analysis should be done on the basis of goals set, and these goals should not be based on sprint work alone.
  11. Count the number of goals met/exceeded/far exceeded. This matrix can then be used to calculate performance ratings in a transparent way.

Scrum is hard, and we have just made the appraisal process more difficult and demanding by using it. If used and adapted as necessary, however, this process should take care of various issues we encounter in the appraisal process. Just as in other Scrum endeavors, the mentor/product owner has a huge role to play, as we are making him or her the owner of the performance of the individual. Going Agile, though, should keep these goals aligned and realistic with the business and its changes, and hence should give individuals realistic chances to grow and measure their annual performances.

To summarize, we are acknowledging that business changes, and performance goals must change accordingly. Making the goal evaluation cycle iterative and adaptive makes the outcome more predictable — and, indeed, makes the process possible.

Article Rating

Current rating: 4 (1 ratings)

Comments

Harish Kandiyil, CSM, 3/20/2012 9:56:22 AM
Arpit - Thanks for sharing your ideas on this topic. Yes, It is very difficult to device any good framework to measure the individual performance in an Agile environment. Most of the Organizations may be following the old customs and rituals. Your ideas and thoughts are excellent pointers for those who prefer to change the system.
Arpit Gautam, CSM, 3/22/2012 1:00:51 AM
Thanks Harish! I started thinking about this a while back. Its really difficult to tell people how evil these appraisals can be in the long run for the morale of an agile team.
Manish Gupta, CSM, 3/23/2012 12:46:41 AM
Your article does add a different dimension to the topic. Having said that, I feel appraisal many a times is a private affair and management does not want to open it up. Secondly, it would be too laborious to handle such a precise and elaborate system. And hence difficult to implement this.
Arpit Gautam, CSM, 3/25/2012 2:43:18 PM
Thanks Manish! I witnessed, in most organizations appraisal is not an incremental and adaptive process, so the idea is to try to make it as incremental and adaptive as possible. And you are right, mostly management does not want to open it up too. Appraisals must not be a closed and private affair for any organization and this article touches the idea of making it more open and collaborative. The approach suggested is laborious and hard to follow just like scrum.But if we start walking on these lines, we can retrospect it, refine it and make it more transparent for everyone's good.
Sowjanya Prattipati, CSM, 3/26/2012 12:35:42 AM
This is indeed a different view. But Scrum is a team work and when you set up goals for each indivicual, it will definity divert him from achiving overall sprint tasks and he will be more inclided to achieve his own goals. I personally feel an individual should be accessed by his contribution towards sprint goals and peer reviews in a scrum team.
Arpit Gautam, CSM, 3/26/2012 1:09:10 AM
Sowjanya, Thanks for your comment! Can't agree more to your point. These ratings and appraisals are evil and anti agile but they do solve a business problem, apparently in a wrong way.They do kill team spirit too. That's why we must not rate individuals on basis of their sprint goals. We must create goals that are more relevant to their own growth and then let them compete to grow. As far as sprint goals are concerned, scrum does a fantastic job for enabling people to fulfill them.
Uday Shete, CSM, 3/29/2012 1:27:18 AM
My two cents. SCRUM by virtue of it is simple and workable solution for complex application development. We focus more on Individuals and interaction over process and tools. And here trying to propose really a big process which includes even product owner who might be from your customer. Appraisal in internal process customer least worried about the same.
The point you are trying to address is ground reality and believe shall be addressed in more simpler way to fit it into SCRUM.
Arpit Gautam, CSM, 3/29/2012 2:14:41 AM
Thanks Uday! I believe appraisal process followed by most organizations is mirror of what they are or used to follow, a waterfall model. Agility should be infused into it.My core point is to make appraisals iterative,adaptive, with timely feedback and unambiguous exit criteria for every goal/story. To overhaul appraisals, Scrum can be used as an inspiration if not as a framework.
Hoa Luong, CSM, 4/6/2012 5:03:07 AM
Thanks Arpit for such a good article. This is exactly what we are doing at the moment for the performance appraisals. In fact, as results of the Retrospectives meeting, we have goals for Team at the end of every sprint. Then members voluntarily select goals they want to archive for next sprint(s). Everything is open and transparent. This makes our personal performance reviews much easier.
Arpit Gautam, CSM, 4/8/2012 4:23:42 PM
Thanks Hoa!I am working on similar lines, trying to propose changes to the process we have. It would be great if you can share few details about the process you guys are following with me.Let me know if I can send you a mail with few queries I have
Michael James, CST,CSP,CSM, 4/10/2012 11:50:50 AM
Other than probationary situations or upon employee request, performance appraisals are counterproductive to their intended aims of alignment, feedback, motivation, fair compensation, and legal protection. It's time for teams to decline to participate in them, and supervisors to decline to perform them. As far as I know, no one's ever been fired for refusing to be subjected to this harmful practice, or for refusing to subject others to it. I think I've had one performance appraisal in the past couple decades.
Arpit Gautam, CSM, 4/11/2012 2:30:22 AM
Good to hear at least someone is doing unorthodox but right thing. Thanks Michael!
Alan Dayley, CSM,CSPO, 8/3/2012 11:04:51 PM
Ugh. No. And again, no.

If you measure individual performance you give them incentive to give you what you measure, at whatever cost, and that includes the cost of breaking the team. Period.

If you measure and compare teams the teams will have incentive to give you what you measure, at the cost of breaking your inter-team organizational agility. Period.

According to the Agile Manifesto, two roles appraise the performance of the team, and only two. 1) The customer decides if the product is acceptable and fulfills their need. 2) The TEAM appraises themselves every retrospective.

An appraisal of an individual that follows the Agile principles would be: 1) The customer of the individual's work, the TEAM, judges if the work is acceptable. 2) The individual appraises himself or herself on a regular basis to see how they can improve using whatever system of self-improvement they like. (A mentor or coach would be good if the individual wants that. And that personal mentor would help the individual with goals and priorities selected by the individual.)

This article says teams can self-organize but individuals aren't allowed or capable of doing so. Well guess what, if the individuals are not allowed to self-organizing their own work, you will have trouble getting teams to do so! And if the individuals you hire are not willing to appraise themselves and improve, you need to hire different individuals, not impose mandatory goals and measures on them.
Alan Dayley, CSM,CSPO, 8/4/2012 2:20:43 PM
Based on some questions I received outside this forum, I'd like to clarify a couple of points in my comment above. First, I said "According to the Agile Manifesto, two roles appraise..." The Agile Manifesto does not define any specific roles. Perhaps I should have said there are two relationships where appraisals occur, that of the customer appraising the product and the team appraising itself. Secondly, the individual's goals are set by the individual and are the type of self-improvement goals anyone should be doing to stay relevant and better contribute in their work.
Peter Jetter, CSM, 10/22/2013 1:30:21 AM
@Michael James, CST,CSP,CSM, 4/10/2012 11:50:50 AM
" As far as I know, no one's ever been fired for refusing to be subjected to this harmful practice, or for refusing to subject others to it. "
Lucky you :-) I refused to participate in a "stack ranking scheme". HR said, that they consider a "letter of reprimand" for me and all team members would be considered "needs improvement". This was backed by leadership team. So as a team we decided following the process is still not good, but the best available alternative.

You must Login or Signup to comment.