Inspection to find defects is waste, inspection to prevent defects is essential.
— Shiego Shingo
Powerful statements like that one spurred a huge revolution in the manufacturing industry. Although we are trying to learn from it, we are still fairly far from preventing defects.
Though we do prevent defects, to what extent? Unit test automation with test-driven development (TDD) is a good example of defect prevention: As we fail, the tests fail, so we make sure the test passes through code. Why was it successful? Because a small unit is built (tested and developed) with a black-box scenario mindset. All scenarios are automated before development.
Challenges with the TDD approach
Though TDD is extremely valuable, it's micro-focused — one method at a time. Though unit code coverage is high (around 100%), not all scenarios are thought through and built.
An alternative is acceptance test-driven-development (ATDD). By definition, acceptance tests are created by the customer, the business, and the engineering team together. This is a great approach to understanding the feature and product well.
Challenges with the ATDD implementation
Is ATDD implemented in the way it is defined? Let's talk about the multiple challenges of ATDD.
- Customers and product owners (POs) often don't get into a discussion with the team about story acceptance. When done, it flows from the PO to the team, and they are primarily business cases.
- Now that the PO defines the high-level business scenarios, test design is completed by the team in a traditional way. One (e.g., tester) writes and the other (e.g., developer) reviews. This is no different from writing traditional test cases. The focus is primarily on the acceptance criteria and a few negative scenarios. All failure cases are not identified. How can you overcome this challenge? Through exploratory test-driven eevelopment (ETDD).
What is exploratory testing?
Exploratory testing is the simultaneous learning, design, and execution of a test. The team as a whole learns the feature, builds test scenarios, and executes them as they build more scenarios. This is one of the most effective techniques of software testing. The learning and designing of tests happens by using the brainstorming technique.
What is ETDD?
ETDD is an approach to building high-quality software by using exploratory testing. In ETDD, exploratory testing drives software design and development.
6 Steps to ETDD
ETDD is a six-step process to delivering value to the customer.
Step 1: Choose the story.
Pick a new story that you want to implement as the next priority item from the backlog. The PO defines the business scenarios.
Step 2: Brainstorm exploratory scenarios.
Brainstorming is a group technique wherein a group of people meet to generate ideas around a specific area of interest.
Before the team designs and codes, it brainstorms (with the customer and PO) to identify all scenarios.
Brainstorming happens within specific guidelines, as defined below.
Step 3: Build the feature.
- Establish space and purpose. Identify a quiet work space. Have clarity in the feature and story.
- Create comfort. Create a comfortable situation so that the team can share ideas.
- Facilitate. Have one of the members facilitate the session. This is a rotational role.
- Create a team. Create a team of 4 – 6 cross-functional members (development/QA/PO/business/customer).
- Share ideas or pass. Every person shares a test idea, one and only one at a time. Share the best idea when it's your turn. Rotate this step. Say, "Pass" if you don't have one.
- Actively listen. By actively listening, you get new ideas from the other person's idea.
- Run until all "Pass." Run the brainstorming session until all team members call out pass twice.
- Do not debate; prioritize. Every idea is a good idea. Prioritize the ideas based on requirements.
The team can build a story by using TDD. Because all scenarios have already been identified, it is easy to start with the basic scenarios and later move toward difficult ones. Make sure that all identified scenarios are reviewed during the development of the story so that no bugs are leaked.
Step 4: Conduct an exploratory test.
Run all scenarios, and make sure that they pass against the following guidelines.
Step 5: Refactor the code.
- Pair test: Create a pair of tests to get thinking more during story testing.
- Design and execute: Simultaneously design and execute in parallel during the test.
- No distractions: Execute in a nondistractive environment. The focus is on story testing.
- Fix and test: Found defects are fixed immediately and tested again. Identify the end-to-end scenarios to automate as part of this story, and then automate.
When the story is done with zero defects, reexamine the code for refactoring opportunities. Any refactoring required, do it, run automated tests, and pass them.
Step 6: Demo the code, and secure sign-off.
Give a demo to the PO and stakeholders, and get a sign-off.
How is ETDD different from exploratory testing?
In exploratory testing, the team learns, designs, and tests the scenarios simultaneously but only after building the feature
. The defects are already injected into the system. In ETDD, scenarios are generated through brainstorming sessions before design and development of the story
. ETDD is more efficient than exploratory testing on its own.
ETDD is a better testing option for the following reasons:
- The team identifies the scenarios before the design and code are written. The scenarios help developers to create good design and clean code the first time — a defect-prevention technique.
- The brainstorming technique allows the team collectively (instead of individually) to share the scenarios. Therefore, all the edge cases and integration points are found from the onset instead of after implementation.
- Team's brainstorming discussions generate product knowledge for the team.
- ETDD helps the team better implement TDD, with a thorough understanding of the features.
- A focused, exploratory test execution allows the team to find all defects before or at least during the development cycle. Fixing these defects is a lot easier and less expensive at this stage.
ETDD also justifies all four Agile Manifesto items:
- Individuals and interactions over process tools — Creates opportunity for the whole team to interact, in every story.
- Working software over comprehensive documentation — When the story is done, it's a high-quality working feature that can potentially be shipped.
- Customer collaboration over contract negotiation — PO and customer involvement in brainstorming sessions fosters collaboration.
- Responding to change over following a plan — Requirement clarity is realized during the brainstorming. Changes in requirements are identified immediately.