The world of software testing has been changing, in some ways radically and in other ways slowly but surely. Almost certainly the biggest change is due to the movement of many organizations toward adopting Scrum1 or other short-cycle iterative approaches to developing software. In my experience, many organizations don't facilitate testers in making this transition and often are not sure how to do so, and therefore many testers are left in the lurch. In this article I'll recapitulate the central practical issues in testing, regardless of the development method, and then sketch the reasons why testers will benefit as they understand and welcome Scrum adoption. There is an auxiliary purpose of explaining to organizations how to harness the skills of testers and garner their support in making the transition to Scrum.
Testing: The central issue
Testers usually don't have sufficient time to thoroughly test a module or functionality.
On examination, the above statement begs two questions:
What is thorough testing?
How much time will be just about sufficient/insufficient?
Take a look at the flowchart below, which represents a procedure implemented in code.
It is not unusual to have procedures/code sections of significantly greater complexity than what is depicted above, running in production systems. To test this procedure exhaustively, we have to consider all combination of flows, with all sorts of inputs. The total number of possible flows through this procedure turns out to be in the region of 1014
If one could write and execute one test case per millisecond, covering all possible flows would need about 3,000 years, give or take a couple of centuries.
Clearly it would be unusual to find this sort of time to test the procedure exhaustively. Most testers would think about 100 test cases to be sufficient to test this thoroughly (assuming that "thorough" is practically the same as "exhaustive"). A set of 100 test cases is nothing more than a negligible, almost infinitesimal, fraction of the total test cases possible. Therefore, assuming that thorough testing is the same as or almost as good exhaustive testing is completely mistaken.
So does it mean that 100 tests are of no use, because this is about one in one million million of the possibilities? How many tests and therefore how much time is sufficient?
The answer is that it depends on which 100 or so tests we select out of the 1014
possibilities. Test-case writing is essentially a selection problem.
For a decently designed, thought-out, and competently coded procedure, the vast majority of flows will be correct; i.e, the vast majority of tests will pass. A tiny percentage of test cases failing may still mean a very large number of bugs. It is this tiny percentage of possible failures/errors that we must detect, and our selection must have the best possibility of doing just that!
The corollary is that there will never be "sufficient"
time for thorough testing as we commonly think of it. Testers have to use a reasonable amount of time to make the best selection based on various criteria to minimize the number of bugs that are left undetected.3
This means that testers have to use various perspectives and be alert for all kinds of signals to make an intelligent selection (all at once or, preferably, iteratively4
) that has a high possibility of exposing the greatest number of bugs.
Perspectives are the result of visibility, and visibility
is what Scrum is about. Working in a cross-functional team with other types of project team members, developers, analysts, interaction designers, etc., talking to customers (or representatives), and participating in sprint-end reviews (which can have a large external stakeholder presence) all help testers gather perspective. They move from being mere black-box testers to all-around expert testers, facilitated by working in Scrum teams.
Now let's examine other problems that testers commonly face.
Problems that testers frequently face
Untestable application and/or lack of setup
Not knowing what is new or has changed
Some bugs block other testing
Untestable application and/or lack of setup
During the early days of the testing cycle, testers can quickly face a situation in which they are unable to execute test cases simply because the basic functionality doesn't work, or there is no easy way to set up the data (i.e., state) so that testing can proceed. In some mature organizations there are smoke/sanity tests, but, again, smoke tests may not pass. Therefore there is a lot of to-ing and fro-ing between the testing group and the programming teams to just get the basic functionality to run in a nondevelopment environment. Often the problem is lack of easy means of setting up the state so that the tests can be run.
Not knowing what is new or has changed
The separation of the programmers and testers into different groups/functions means that there is a greater possibility of communication gaps. Programmers have been told of changes to the originally described functionality, but the testers don't come to know of it, and they test the functionality based on the older/initial understanding of what was needed. They could, in the worst case, go off on tangents or wild-goose chases; in the best case, they could report as bugs some behaviors that are really not so.5
Some bugs block other testing
A functionality that is not working could restrict setting up of data or block access to other functionality that needs to be tested. To unblock this requires a report and at least one handover.
Programmers may think of themselves as a special breed, as the ones creating software and thinking all other roles are just support roles.6 Many of them like to think of their output as unmarred by any blemish. Therefore they do not like other people to point out errors and may even believe that third-party testing is a waste of time, at least as far as the individual programmer is concerned. These views are the underlying reasons for programmers to adopt a noncooperative stance toward testers. This is particularly easy to do, since the others in their group (i.e., other programmers) will have similar attitudes, to a degree. Also, as the testers are in a separate group, which in many organizations is deployed in some sort of competitive stance,7 testers can easily get painted as the villains. Testers are often used to this situation and think of it, if at all, as an occupational hazard.
How the problems outlined above are solved or largely mitigated in a good Scrum adoption
Considering each of the above problems in turn:
• Untestable application and/or lack of setup
Working as a part of the same team, you get a chance to look at the new code working on the development machine. You can execute a few tests on the development machines, during which time you learn a lot about data setup for relevant user stories/scenarios. Following this you can personally facilitate the build and deployment to a team server or the testing server. This can help you develop an insight into why the software that works on one machine fails on another. It also gives you an advance view into integration problems. I would encourage you to build all or some of the software yourself, to help the team with creating the shippable product increment.
• Not knowing what is new or has changed
This is will be rare when you are in the same team and are part of the shared commitment, and also in sync with the programmers due to the daily stand-ups.
• Some bugs block other testing
You can work with developers to fix the offending bug. However, if at first you find this daunting, you can work with developers to reproduce the bug, which is often a big help to the developers. You still might have access to other developer machines, so you could continue testing on other functionality.
• Developer intransigence
Most programmers are human. Yes, really! So it's difficult for them to be belligerent when all of you are on the same team and everyone shares the same commitment.8
Role of testers in Scrum teams
While the previous section explained why adoption of Scrum can make the life of a tester easier, this section will elaborate on how testers can contribute to the success of a Scrum team.
Find bugs early.
Early discovery of bugs contributes dramatically to the efficiency of development. Basically, late detection of bugs results in disproportionately greater turnaround times to fix those bugs. Working in the same team as the programmers and analysts provides excellent opportunities for testers to find bugs early.
Work with the whole team to learn and teach (functionality, testing, and programming).
Software development depends on people working on the project to learn about various facets: functionality, tools, programming languages, testing, etc. A group that is made up of diverse individuals with different skills can form a great learning platform, and much broader learning is possible, than if you are in a homogenous team. So your role is to learn from as well as teach the team.9
Increase visibility for any other independent testing group (more on that later. . . .).
The testing group in many Scrum projects is loosely formed. There are many testing activities that might have to be done outside of the Scrum team's efforts. These include security testing, load and stress testing, and platform compatibility testing as well as integration testing. Ideally these are also to be done as a part of the sprint activities. However, some or all of these might be done in a separate stabilization sprint, or there could be a separate specialist testing group that takes care of these activities, or testers who are part of Scrum teams can be doing these activities outside of the sprints.
In any case, these activities require some information as to how development was done and the list of bugs detected already. This early visibility can be provided by testers who are within Scrum teams.10
Generate validation questions.
The customers or representative (analysts) are concerned with a work flow or application flow for automation. They concentrate on the usual flow or outcome, also known as the "happy path," in user-story parlance. This is natural for them, but it often results in alternate flows being missed out and even sub-user stories being left out and discovered much later at awkward junctures. However, a skilled tester with functional knowledge can identify unusual what-if scenarios (precipitated by input combinations or other back-end conditions) that were not thought of and that the system therefore will not be able to handle gracefully. Identification of such scenarios is useful for the whole team and for the customer.
Role of test leads and managers in the Scrum world
In this section we will see how test leads and managers can contribute and play a critical role in enabling testers as well as bringing them to a higher level of maturity.
They enable testers to become expert testers.
Test managers and test leads should primarily be taking steps so that the testers for whom they are responsible become better testers continuously and move toward higher levels of expertise. Many testers describe themselves as QA, while they actually perform a QC role. Working with Scrum teams provides good opportunities to carry out true QA activities as well as instilling an attitude that quality is everybody's business.
They remove impediments to the effective functioning of testers.
Testers in their day-to-day work will come across many obstacles, and their efficacy significantly increases when these obstacles are tackled as quickly as possible. Test leads and managers can be a great help in this. While the ScrumMaster is present in this scenario, changes and obstacles that have to do with testing activities in particular might be more effectively taken up by test managers.
Coordinate testing strategy and serve as a sounding board in a large group.
In large groups, to roll out a product requires a lot of coordination within the testing activity itself. Traditionally, testing cycles were planned with a view of providing a quality gate, and to ensure that the outcome of development had been tested "thoroughly." As we have seen, "thorough testing" is often not very meaningful; and testing with limited resources11 to cover unlimited possibilities (1014 is as good as unlimited) is basically a guess. The more informed a guess the better. Therefore, testing activity done within Scrum teams and the results thereof provide valuable information that can shape testing strategy, particularly regression testing. This advance information helps identification of modules/sections/areas requiring greater or lesser testing, as well as what kind of testing. Further learning of a given release can be gathered centrally and used for the benefit of all (programmers, analysts, customers, and testers) for the following releases. Essentially this means that information collected earlier will guide the decisions that will be taken to make the best use of resources with the aim of improving the effectiveness of testing. In yet other words, this information helps makes iterative selection of test cases to be executed possible on a larger scale.
Testers are often much less effective than they can be. This is frequently due to the project's organizational structure, and it is also due to the manner in which projects are executed. This leads to fairly low-level work for testers most of the time. Essentially, their potential is often unused.
The Scrum approach has the potential to mitigate most problems arising from these factors and, in turn, contribute to the overall success of the project. The organization of people in cross-functional teams might initially feel uncomfortable for testers. However, overall it is beneficial in both the short term (greater effectiveness) and the long term (improved ability). Testers' involvement in cross-functional teams leads to their improvement as professionals, with wider ranges of exposure and responsibilities.
This article is the result of my consulting experience with a few project groups over the last three years or so. I've seen many of these problems over and over, as well as the misunderstandings and opinions embedded in the minds of many a team member and manager.
Or "Agile" development; the author prefers to avoid that word, since in the vast majority of the cases it is used as a meaningless buzzword.
Pressman R. "Software engineering: A practitioner's approach."
We may never be able to tell exactly how many bugs are undetected; even estimating this requires a significant effort.
Iterative selection, based on results of an earlier selection, is the approach taken by exploratory testers.
One might think that changes are communicated to everyone, but these changes can either be informally communicated or stressed only to those who need to know of it urgently (i.e., programmers). Another possibility is that the clarification the programming team asks of analysts will only be communicated back to the questioners.
There is undoubtedly some justification for this view, mainly because there have been significant programs developed single-handedly by individual programmers.
The phrase "creative tension," to put a positive spin on this type of arrangement, is not unknown.
Refer to Marick's article.
On a personal note, a couple of stints as a tester certainly made me a better developer; I've seen this work in the reverse as well.
Bugs discovered by the Scrum team within the sprint time period may not be recorded in a bug-tracking system. These have been found out, informally reported/demonstrated, and fixed by team members in their
drive to meet the sprint goal/commitment.
By "resources," we are not referring to humans of any shape or size.