One of the myths of Agile software development is that documentation is not required or useful. It is true that one of the core values within the Agile Manifesto is "Working Software over Comprehensive Documentation." However, note the word "over" in this statement. The Manifesto is not saying no documentation; it's saying there is a preference for working software over documentation. The goal is to remove impediments from the system and leave things that add value.
If your organization is creating lengthy documents to produce software and you're still struggling to release software on time and within budget, then ask yourself this question: "What value are the documents adding?" The value question is an essential part of Agile as well as of Lean Thinking. Lean Thinking was popularized by Toyota and has been widely adopted across many industries. Its premise is to eliminate waste from the system and get down to the essence of what it takes to drive value to the customer. It also focuses on self-organizing and self-correcting teams to drive quality and efficiency in the system.
The concept of Agile requirements definition and management (RDM) is not new. However, the struggle to figure out how traditional requirements cycles fit within an Agile framework remains convoluted. For a system to work, an organization needs to think about the entire lifecycle, not just about the software development portion. If you start at a high level within your company and analyze what documentation is being produced in order to create finished code, you'll most likely find that more than 50 percent of the documentation created was not used. Why are we creating waste?
One theory relates to the complexity factor. As a system starts to mature, it starts to move toward complexity. It becomes increasingly difficult to understand, organize, manage, and maintain these systems. People have a tendency to add more to the system to help fill gaps and create protocol to maintain and control it. As organizational turnover occurs and the rules, regulations, and business climate change, the system must also adapt. This causes the complexity of the system to grow exponentially, until it gets to the point where it's difficult for it to function properly, or it's actually restrained from doing so. This is precisely why the best systems — or, for that matter, the best products — are simple and streamlined. This is exactly why Lean Thinking removes waste from the system. Requirements definition and management is no different in its vulnerabilities to the complexity factor. This is why companies sometimes end up with requirements specifications that are hundreds of pages long and extremely difficult to follow and manage.
If you're looking to adopt Agile and want to run a leaner operation, you have to take a holistic view of the organization. Requirements definition in Agile has to be looked at through a separate lens, not strictly in conjunction with the development team. However, the same principles used in Agile software development can be applied to RDM as well. Let's look at a quick example of how this works.
How it works — and doesn't work — in Scrum
The most popular of the Agile frameworks in use is Scrum. Other frameworks include XP, Crystal, and Kanban, to name a few, but since Scrum is the most commonplace, we'll use it for demonstration purposes. (It's important to mention, however, that some of these frameworks can be combined. For example, portions of Kanban can be used within Scrum to control team capacity constraints by limiting work in progress.)
Scrum allows development teams to build software incrementally over two- to four-week events, or sprints (see Figure 1 below). Requirements are fed into a product backlog prior to sprint inception; they're decomposed into sprint backlogs items through sprint planning. The development team starts by discussing what needs to be developed in a given sprint based on organizational needs and strategy. The work items are pulled from the product backlog and directed by the product owner, with the process controlled and managed by the ScrumMaster. The goal for the business is to make sure they feed the product backlog and can support and describe what needs to be built by the development team prior to the start of the sprint.
The problem is that most organizations struggle to keep pace, and/or they don't have the right level of detail defined in the product backlog to properly tie into the development sprints.
Figure 1: Scrum/Sprint
Agile RDM steps in
Agile requirements definition and management (RDM) was specifically designed to solve this problem by outpacing the development team. In other words, feed the product backlog faster than the development team can produce code. The framework can be used for just-in-time requirements definition or to build a repository of requirements for future use. In either case, if a team is using Scrum, it's working from the product backlog. Since the product backlog is a "backlog" of work, the required pace of filling the backlog is driven by the designated Sprint time frame. The goal is to make sure that the business (specifically, the product owner) can clearly articulate what needs to be built and define what is of high quality. To accomplish this, the requirements cycle follows a Scrum-like process that mirrors the development cycle but stays two to three steps ahead (see Figure 2 below). The goal is to create a process by which requirements can be thoroughly vetted, organized, and communicated in a manner that is iterative, timely, and quality-focused.
Figure 2: Agile requirements definition and management (RDM)
It all starts by identifying and filling a requirements backlog. This type of backlog is a list of items that need to be defined in order to fill the product backlog. The end goal could be user stories, visualizations, functional requirements, and so on. Using requirements planning and prioritization, the requirements team decides, based on the business strategy and objectives, what needs to be defined and built. Like the development team, the requirements team plans its sprint, performs the work, and reviews the outputs. If the outputs meet expectations, then they can be moved to the product backlog. In many cases, organizations have documents that need to be created to pass certain "tollgates," or organizational milestones. These items can also be put in the requirements backlog, but they may not end up in the product backlog. Instead, such documents often become reference materials for the development team to use. This is where traceability from the product backlog to any external documents becomes important in establishing project continuity.
Another important portion of RDM is called decomposition. Decomposition is the process by which the product backlog items are communicated and refined in collaboration with the development team. Decomposition can be used in several ways. One is to set up a culture of collaboration in which the development teams are brought into the requirements phase to refine the product backlog. In Scrum, this is commonly referred to as "grooming the backlog."
Another way to use decomposition relates to procurement and/or timing delays. Some projects experience gaps in time between when requirements are defined and when development starts. There are many reasons why this happens, but it's mainly important to note that this occurs on a regular basis. The larger the gap in time between definition of requirements and development, the more the risk that occurs with developing the right product. The longer the gap, the higher the risk of losing vital team members, knowledge, and overall team availability. In this case, decomposition is a way to pick up where things left off, by using the product backlog to communicate and share requirements.
Agile is quickly becoming the most popular way of developing software because it fosters continuous improvement, time-boxed development cycles, and more quickly delivering value to the end users. That value will be driven to a large extent by the quality and clarity of requirements that feed the software development process. An agile, lean, and timely approach to requirements as the starting point will help to ensure that the process is optimized.
There are many flavors of Agile on the market today, and I've discussed but a few of them in this article. The key is to figure out what works for your organization, and then start experimenting. The faster you dive into trying to be more Agile, the faster you will start seeing the benefits it brings.