Get certified - Transform your world of work today

7 Ways to Ensure Automation Success

08/01/2013 by Steve Gibson

We're all aware of the importance of test automation in Agile development. In fact, test automation encompasses three of the four pillars of proper Agile testing (automated acceptance testing, unit testing, and automated regression testing) (Robert Walsh). Think of automation as a quality safety net: Without it, you can't safely refactor the application, and your team is stuck running a set of ever-increasing, boring, manual tests, over and over. That's not sustainable.

I've seen test automation done correctly, and I've seen it done poorly. Leveraging my experience, I present to you my top seven recommendations for ensuring that your automation effort is successful.

Test automation is more than just UI automation. Mike Cohn's automation pyramid illustrates two layers beneath the UI that also demand attention:  API tests and unit tests. UI tests are costly to build, inherently more fragile, and generally can only be completed toward the end of a story's development. API and unit tests occupy the majority of the pyramid's real estate, and so they should reflect a commensurate ratio of your Scrum team's automation tasks. Plus, they can be built before the code reaches the UI, so dig in already!

Do not use record and playback automation. Ever. Those aware of the difference between selenium and the selenium IDE will tell you the same thing. Record and replay tools generate "disposable" automation that is extremely fragile and unmaintanable. At the speed of Agile, a team cannot afford to spend time fixing a bunch of fragile tests.

Automation demands a whole-team approach. The worst thing you can do is to designate someone "the automation guy/gal." Without a sense of team ownership, you'll never maintain automation and feature parity. Weave test automation goals into your scrum team's Done-ness criteria so it's everyone's responsibility. See Lisa Crispin's book Agile Testing for more details on this (you should have already read it).

Develop a robust automation framework.  (This is the most important piece of the automation puzzle.) You're not using record and replay, are you? Good! But now you're faced with the problem of developing the automation framework. Your framework needs to be designed to solve problems like: seeding data for your tests, cleaning up test artifacts, results collation and reporting, concurrency, etc. Build your framework incrementally using Agile principles, just like you would any new feature in your application under test. Deliver value fast. Don't try to architect for the future. Sound familiar?

Automate the automation. Run your automation all the time: unit tests before commits, the rest after deployment to a test environment (generally). Use a scheduler such as Jenkins to poll your source control and/or key off of code-deployment builds. The fewer humans involved in this process, the better.

Be transparent. Use information radiators to communicate the build results to your teams. Make the results easily digestable to passers by: red means failure, green means success. It's as simple as that. Show trend graphs to give added insight.

Stay on top of the failures. Tests can fail for any of three reasons: test fragility, environment issues, and/or application/code problems. Not staying on top of the failures means that your tests won't be highly regarded, and most likely they'll end up being ignored. Test runs that stay "red" all the time get disregarded. What a waste!

These are just seven recommendations. If you want the rest of them, find me on LinkedIn ( or shoot me an email (