Reducing the Test Automation Deficit

How a test automation backlog can help repay the debt

6 February 2008

Many companies with existing legacy code bases bump into a huge impediment when they want to get agile: lack of test automation.

Without test automation it is hard to make changes in the system because things break without anybody noticing. Defects aren't discovered until the new release goes live; worse, they are discovered by the real users, an embarrassment that results in expensive hotfixing, or even a chain of hotfixes, as each hotfix introduces new, unanticipated defects.

These risks make the team terribly afraid to change legacy code, and therefore relucant to improve the design of that code, which leads to a downward spiral in the quality of code as the system grows.

What to do about it

Your main options in this case are:

  • Option 1: Ignore the problem. Let the system decline into entropy death and hope that nobody needs it by then.
  • Option 2: Rebuild the system from scratch using test-driven development (TDD) to ensure good test coverage.
  • Option 3: Start a separate test automation project where a dedicated team improves the test coverage for the system until it is adequate.
  • Option 4: Let the team improve test coverage a little bit each sprint.

Guess which approach usually works best in my experience? Yep, the last one - improve test coverage a little bit each sprint.

The third option may sound tempting, but it is risky. Who's going to do the test automation? A separate team? If so, does that mean the other developers don't need to learn how to automate tests? That's a problem. Or is the whole team doing the test automation project? In that case their velocity (from a business perspective) is zero until they are done. So when are they done? When does test automation "end?"

No, it's better to improve test coverage a little bit each sprint. The question is, how is that accomplished?

How to improve test coverage a little bit each sprint

Here's an approach that I like. In summary:

  1. List your test cases.
  2. Classify each test by risk, how expensive it is to do manually, and how expensive it is to automate.
  3. Sort the list in priority order.
  4. Automate a few tests each sprint, starting from the highest priority.
Step 1: List your test cases

Think about how you test your system today. Brainstorm a list of your most important test cases--the ones that you already execute manually today, or wish you had time to execute. Here's an example from a hypothetical online banking system:

 

Test case 

Change skin 
Security alert 
See transaction history 
Block account 
Add new user 
Sort query results  
Deposit cash  
Validate transfer  

 

Step 2: Classify each test

First classify your test cases by risk. Look at your list of tests. Ignore the cost of manual testing for the moment. What if you could throw away half of the tests and never execute them? Which tests would you keep? To answer these questions you would have to consider both the probability of failure and the cost of that failure.

Highlight the risky tests, the ones that keep you awake at night.

 

Test case 

Risk

Change skin   
Security alert   
See transaction history   
Block account   
Add new user   
Sort query results    
Deposit cash    
Validate transfer    

Now think about how long each test takes to execute manually. Which half of the tests take the longest? Highlight those.

 

Test case 

Risk

 Manual Test Cost

Change skin     
Security alert     
See transaction history     
Block account     
Add new user     
Sort query results      
Deposit cash      
Validate transfer      

 Finally, think about how much work it is to write an automation script for each test. Highlight the most expensive half.

 

Test case 

Risk

 Manual Test Cost

 Automation Cost

Change skin       
Security alert       
See transaction history       
Block account       
Add new user       
Sort query results        
Deposit cash        
Validate transfer        
Step 3: Sort the list in priority order

So, which test do you think we should automate first? Should we automate "Change skin" which is low-risk, easy to test manually, and difficult to automate? Or should we automate "Block account" which is high risk, difficult to test manually, and easy to automate? That's a fairly easy decision: "Block Account."

But here's a more difficult decision. Should we automate "Validate transfer" which is high-risk, hard to test manually, and hard to automate? Or should we automate "Deposit cash" which also is high-risk, but easy to test manually and easy to automate? That decision is context dependent.

To sort the list, you need to make three decisions:

  1. Which do you automate first? The high risk test that is easy to test manually, or the low risk test that is difficult to test manually?
  2. Which do you automate first? The test that is easy to do manually and easy to automate, or the test that is hard to do manually and hard to automate?
  3. Which do you automate first? The high risk test that is hard to automate, or the low risk test that is easy to automate?

Those decisions will give you a prioritization of your categories, which in turn lets you sort your list of test cases by priority. In my example I decided to prioritize manual cost first, then risk, then automation cost.

 

Test case 

Risk

 Manual Test Cost

 Automation Cost

Block account       
Validate transfer        
See transaction history       
Sort query results        
Deposit cash        
Security alert       
Add new user       
Change skin       

That's it! You now have a prioritized backlog of test automation stories.

You could of course also invent some kind of calculation algorithm. A simple algorithm is that each highlighted cell = 1 point. Then you just add upp each row and sort. Or you can just sort the list manually using gut feel.

You could also use a more specific unit for each category, if my simple binary scheme isn't sufficient.

 

Test case 

Risk

 Manual Test Cost (man-hours)

 Automation Cost (story points)

Block account   high  5 hrs  0.5 sp
Validate transfer    high  3 hrs  5 sp
See transaction history   med  3 hrs  1 sp
Sort query results    med  2 hrs  8 sp
Deposit cash    high  1.5 hrs  1 sp
Security alert   high  1 hr  13 sp
Add new user   low  0.5 hrs  3 sp
Change skin   low  0.5 hrs  20 sp

Remember though that our goal for the moment is just to prioritize the list. If you can do that with a simple and crude categorization scheme then there's no need to complicate things right? Analysis is useful but over-analysis is just a waste of time.

Step 4 - Automate a few tests each sprint

Keep in mind that no matter what, each new product backlog story should include test automation at the story level. That's the XP practice known as "customer acceptance tests." Not doing that is what got your system into this mess in the first place.

In addition to implementing new stories, we want to spend some time automating old test cases for other previously existing stories (the ones in the tables above). So how much time do we spend? The team needs to negotiate that with the product owner. The agreement will typically take one of the following forms:

  • "Each sprint we will implement one test automation story"
  • "Each sprint we will implement up to 10 story points of test automation stories"
  • "Each sprint we will spend about 10% of our time implementing test automation stories"
  • "Each sprint we will finish the product backlog stories first, and then spend the remainder of the time (if any) implementing test automation stories"
  • "The product owner will merge the test automation stories into the overall product backlog, and the team will treat them just like any other story."

The exact form of the agreement doesn't matter. You can change it every sprint if you like. The important thing is that the test automation debt is being gradually repaid, step-by-step.

After finishing half the stories on your test automation backlog you might decide, "Hey, we've paid back enough debt now! Let's just skip the rest of the old test cases, they're not worth automating anyway," and dump the rest. Congratulations!

So we'll no longer have a problem with test automation debt, right?

Wishful thinking. This pattern won't erase your test automation deficit, but it does give you a way to tackle what can be an overwhelming task.


Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.



Article Rating

Current rating: 5 (1 ratings)

Comments

Oscar Lantz, CSM, 2/8/2008 3:12:47 AM
I see the point of implementing automated tests of legacy code a little bit at a time. But what if the biggest impediment is that the legacy system will not easily be test automated. What if the team sees a huge chunk of work that needs to be done to even be able to implement the first automated test? This is what I find to be hard to motivate the team and the product owner to start working on.
Henrik Kniberg, CST,CSP,CSM,CSD,CSPO,REP, 2/8/2008 2:23:46 PM
Good point. Maybe the test automation backlog will show that *any* test at all would be prohibitively expensive to automate. That's certainly useful information, allowing us to make an informed decision about whether or not it is worth paying back the test automation debt, and if so how fast.

However in my experience even the most horrid pile of legacy code usually has *some* little corner somewhere where adding a test or two would be relatively easy.

If you're stuck with legacy code that is really hard to test I can really recommmend reading Michael Feather's book "Working effectively with legacy code".

(Hmmmm... I wonder if <a href="http://www.amazon.com/Working-Effectively-Legacy-Robert-Martin/dp/0131177052">links</a> work here.)

(Or maybe like this: http://www.amazon.com/Working-Effectively-Legacy-Robert-Martin/dp/0131177052)
Anonymous, 4/18/2012 4:24:08 AM
I have a Question.
In a project where we are building every thing from scratch. Should we implement automated test every sprint or when the UI are stable and does not change much (which can be after 4-5 sprints. Or implement some simple automaated test in each sprint whci does not need maintenance and make a seperate sprint for implementing a whole range of test once that UI is stable after 4-5 sprints.

You must Login or Signup to comment.