The road to Agile
Waterfall is a software development model and sequential design
process that has worked for us over the past five decades until now. Let's take a brief look at its history before we discuss the phases in software development.
The first mention of the Waterfall model goes back to June 29, 1956, when Herbert D. Benington, at the Symposium on Advanced Programming Methods for Digital Computers, described the use of similar phases in software engineering. In 1970, the first article on the Waterfall model was published by Winston W. Royce. Royce did not use the term Waterfall
in that article, but he presented the model as an example of a flawed, nonworking model. The earliest use of the term may have been in a 1976 paper by Bell and Thayer. In 1985, the United States Department of Defense captured the Waterfall approach in DOD-STD-2167A, their standards for working with software development contractors, which stated that "the contractor shall implement a software development cycle that includes the following six phases: Preliminary Design, Detailed Design, Coding and Unit Testing, Integration, and Testing."
That is a great start — an efficient model to develop software. It's pure common sense to want to develop a preliminary design to ensure the feasibility of a potential workable solution and elaborate on the design at a later stage. The goal is to kick off development at the right time and ensure that the solution covers all users and test cases and that it has all the necessary features. Because software development itself is not sequential, we integrate these features until the independent pieces work together. We test them one last time before deploying them to production. Beautiful, isn't it?
Let's take a look at the Agile model. What is Agile, anyway? Before we answer, let's take a look at the history of Agile.
Incremental software development methods trace back to 1957. In 1974, E. A. Edmonds wrote a paper that introduced an adaptive software development process. Concurrently and independently, the same methods were developed and deployed by the New York Telephone Company's Systems Development Center under the direction of Dan Gielan. In the early 1970s, Tom Gilb started publishing the concepts of evolutionary project management (EVO), which have evolved into competitive engineering. During the mid to late 1970s, Gielan lectured extensively throughout the U.S. on this methodology, its practices, and its benefits.
A collection of lightweight software development methods evolved in the mid-1990s in reaction to the perceived heavyweight Waterfall-oriented methods, which critics called heavily regulated, regimented, and micromanaged; although some proponents of these lightweight methods contended that they were simply returning to earlier software practices. These lightweight methods included: from 1994, unified process and dynamic systems development method (DSDM); from 1995, Scrum; from 1996, crystal clear and extreme programming (aka "XP"); and from 1997, adaptive software development and feature-driven development. Although these originated before the publication of the Agile Manifesto in 2001, they are now collectively referred to as Agile methods and often abbreviated loosely as Agile, with a capital A, although this is progressively becoming deprecated.1
Agile is an iterative and incremental development model to deliver business value in which each iteration produces a potentially shippable product. From what has been discussed, it is safe to say that we have always found value in the software development life cycle (SDLC) process to bring in some symmetry and governance to the projects we execute, and we never really wish to eliminate them. Below is a diagram to help explain the different phases of SDLC.
If you want to adhere to your organizational compliance and project management office practices, an increment can be called a potentially shippable product only when you perform all of those activities defined in each phase. That means going through a complete SDLC for each sprint, which doesn't sound like a difficult task. However, when you start moving your proverbial pawns, you will find that your king is in a deadlock. You will face certain challenges during the following SDLC activities:
- Development integration testing (DIT)
- System integration testing (SIT)
- User acceptance testing (UAT)
What is the big deal with these activities? When a development team develops a set of features, these features don't get tested against existing and planned features. They go through unit testing to ensure that the functionality supports every element in the acceptance criteria and that all use cases are covered and don't break existing functionality or whatever is captured during unit testing up to that point. Hence, you'll need a stable, up-to-date, and controlled environment to perform DIT.
After the team ensures that the package has successfully completed DIT, it is deployed in a SIT environment to test for its ability to perform the developed functionality in a full-system process. That process requires another controlled production-like environment, and the same applies to UAT. When I say controlled
, I mean that you have a team or department that deals with security, limiting access to stakeholders. By contrast, the development team is granted permission to read but not write to binaries or configuration files, and every request for change during DIT, SIT, or UAT flows through your configuration and release management team, which is also a regulatory requirement in most industries. Remember that a change is not only the one that is requested for production but everything that is requested after development
Below are some of the challenges during DIT, SIT, and UAT activities:
- Infrastructural challenges; availability of environments
- Stakeholder and user availability and engagement
- Change cycle time; to request one in UAT, it should cycle through DIT again
None of the above challenges will actually stop your progress, but the one that will is ensuring that the entire process is performed within the sprint duration. For convenience, let us consider a sprint duration of three weeks. You will normally deploy a package to DIT and progress through the SIT and UAT phases of testing. You will need at least one or two weeks to complete this process. This will reduce your development time to a week or two. But the core of the problem is that you will spend two weeks on testing for a piece that took a week to develop. Sound familiar?
Overcoming the challenges
"What is the solution?" you might ask. What is continuous delivery? "Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time."2
How will we ensure continuous flow of delivery with business value when our release cycle time is three weeks or more? It is only possible if our change process is continuous or automated.
For any enterprise, it is crucial to have a sustainable change process. Sustainability is not what is covered in the process but what is considered simple. After we build our continuous integration engines but before we start development, we should shift our focus to the strategy for CD. Iteration by iteration, we should mature our change process to the level at which we truly "embrace the change." That means adopting one-click deployments, identifying interim/patch and full releases, automating deployments and stakeholder notifications, and ensuring release area preparations upon stakeholder sign-off.
A feature and value-driven change process looks something like this:
- The development team completes unit testing and requests a change list from the Configuration Management team for DIT deployment. This change list typically consists of a single story or more. It is worth noting that this is not actually a potentially shippable product (PSP) but a part of it. Every time a story is completed, a feature is ready, or a bug is fixed, it will go through the test cycle independently.
- Once the development and testing teams deploy, test, and sign off, automated scripts run and deploy the story to SIT, which sends a notification to the stakeholders requesting their sign-off.
- After stakeholders sign off, our release scripts deploy the story to UAT and request users for their sign-off.
- Once users sign off, this package will be marked as production ready and await a click from the release manager to unleash into production.
We should continually improve the automation and maturity of the release process to the point that it requires only a single click by the developer to deploy the change to production. Plan appropriately for fine-tuning your branching strategies in source control and developing automated scripts. This results in not a only quick win but a big win for good.
1. The Info List: Agile Methods
2. "Continuous Delivery," Wikipedia