The systems development life cycle (SDLC) is a conceptual model used to develop information systems with well-defined phases. The phases are: user requirements definition, system requirements definition, analysis and system design, system development, testing, implementation, and maintenance. Some of the models used are Waterfall, spiral, and rapid prototyping. These models are called plan-driven, lying on one end of the "planning emphasis" continuum. At the other end of the continuum are the Agile methods, which are considered adaptive rather than predictive. This paper presents a systems-theory perspective of both models, including the inherent challenges that need to be addressed when transitioning from traditional to Agile methods.
The nature of systems development
Software development is a highly complex activity. It is characterized by variable requirements, the need for specialized and diverse skills, changeable and sophisticated technology used to develop and deploy software, and difficulty in management of the people who deal with such complexity every day. It is not uncommon to find organizations overwhelmed by the inherent complexity in implementing systems development projects.
Therefore, systems development processes can be described as being complex, unpredictable, and poorly defined. In other words, these processes don't have well-defined inputs and outputs and therefore are considered unrepeatable.
The traditional systems development model
Almost since its inception, a rational, engineering-based approach, such as the Waterfall method, has been used to develop projects. This choice seems to have been grounded in "hard-systems thinking," which assumes that problems can be well defined, processes can be optimized, and results can be well predicted. Extensive up-front planning is done to measure and control the variations in the development life cycle.
The essential nature of the traditional software development life cycle is as follows:
- The goals are to thoroughly understand users' needs, craft a solid design, develop software flawlessly, and implement a functional system that satisfies user needs.
- There is a heavy emphasis on thorough planning to deal with risks.
- It is based on the principles of hard-systems thinking — identifying alternate ways of reaching the desired state (S1) from the initial state (S0) and choosing the best way to achieve it (S0 – S1).
- Such an approach assumes that problems are well defined and that an optimum solution can be arrived at by extensive, up-front planning.
- It also assumes that the processes are predictable and can be optimized and made repeatable.
- It is also based on the assumption that processes can be adequately measured and that sources of variations can be identified and controlled during the development life cycle.
- In summary, the traditional software development life cycle is highly process-centric.
Based on the above understanding of systems development, organizations adopt a management style that is:
- Command-and-control-based, with a set hierarchy. Therefore these are predominantly mechanistic organizations geared for high performance in a stable environment.
- Characterized by high formalization and standardization. People with different specializations are assigned roles for producing defined outcomes. In addition to this, they also produce a significant amount of documentation that explains the software and its technical and design specifications.
- Notable in that though customers play an important role, their participation is at maximum only during the specification and implementation stages.
From the customers' perspective, there are pros and cons to traditional approaches. The definite advantage is its scalability. Very large projects and mission-critical projects need a strong plan and close supervision, and they need provision for stability. The shortcomings are based on the assumption that customer requirements are well understood at the beginning and don't change much, which is rarely the case. Another fundamental problem is that processes take too long, such that when end users see the system, many things — including user requirements — have changed drastically.
To summarize, there are two broad themes that emerge in traditional systems development, around which the problems are centered:
- The development life cycle and its processes: The way processes are understood and designed and managed (as though they are well defined, predictable, repeatable processes) is highly process-centric. It uses hard-systems thinking, engineering-based in its approach. The focus is on achieving stability.
- The management style: The way people are organized, managed, and controlled is through "high command and control," a formalized and standardized appropach with limited customer interaction.
To address the challenges posed by the traditional methods, a set of lightweight methods called "Agile" were developed more than a decade ago. They include XP, Scrum, feature-driven development, and more. Agile methods, using various tactics, try to overcome the limitations of the dynamic nature of systems development projects.
Agile methods address the inherent problems of traditional systems development using two scientific concepts: One method is using "empirical process control." One can also say that this is based on "soft-systems thinking." The other method is seeing the systems development as "complex adaptive systems" and designing team structures and methods around that.
Here is how Agile addresses the shortcomings of the traditional software development life cycle:
- Agile methods adopt the empirical process control model as against the defined process control model. Empirical process control is meant for processes that aren't well defined and are unpredictable and unrepeatable. It implements control through frequent inspection and adaptation. Since software development processes are highly complex and variable in nature, empirical process control methods seem to be a best fit to deliver results.
- According to complexity theory, a complex adaptive system (CAS) self-organizes and adapts to changes in the environment without any central rules governing its behavior. An Agile development system can be likened to a CAS, responding to complex and unpredictable changes in the requirements.
- The development model changes from a linear life cycle model to an evolutionary delivery model. This is characterized by short iterative cycles, with periodic reflections and adaptations and continuous integration of code into the overall system under development.
- Since the cycles are short and iterative, they provide the necessary flexibility and speed to adapt to changes in requirements through constant feedback from the stakeholders.
- Agile methods also need constant collaboration with customers, using their input and feedback at various checkpoints during each iterative cycle.
The organization design required is that of an organic management system. Organic design provides for the inherent instability in the systems development projects through less formalization, decentralization, and less supervision. In order to get the necessary flexibility and speed, the Agile projects self-organize themselves into small teams, with a "servant leader" who facilitates the team's processes and goals. The teams are made up of people with diverse sets of skills who often take on different responsibilities as situations arise. The teams directly interact with customers to set goals and plan and implement them. The entire team goes through repeated cycles of thought-action-reflection, with the primary goal being learning and adapting, enabling them to respond quickly to emergent situations. The team produces only the most necessary documentation needed to communicate between the different stakeholders.
What are the main challenges facing organizations in migrating to an Agile environment? The transition will call for changes in organizational structure, culture, and management practices. There are challenges and difficulties inherent in a shift of organizational and management practices:
- The organization must move from a mechanistic style to an organic one, or to an effective mix of both.
- This shift means relinquishing a command-and-control style to a collaborative style. The project manager becomes a facilitator who guides and coaches the team members and coordinates their collaborative efforts. The major challenge here would be to make the manager relinquish the authority he or she enjoyed.
- Organizational culture affects the way individuals behave and work with each other. Culture has significant impact on organization processes such as decision making, problem solving, social relationships, communication, and more. Moving to Agile practices requires a good understanding of the existing processes before changing them.
- Since Agile practices discourage extensive documentation, most knowledge remains in the heads of the team members. This creates a high dependency, for management, on the individuals. This threat needs to be dealt with by finding ways to retain knowledge and people within the organization.
- Reward systems need to change to promoting teamwork and retaining people.
There are also challenges in reorganizing teams:
- Agile practices are heavily dependent on collaborative teamwork. Teams have to be trained and coached to perform as self-organized teams, to learn and adapt to the environment.
- For individuals to work in different functions within the same team requires them to be competent enough. This means that Agile projects need to be staffed with above-average people. This can cause serious staffing and morale issues.
- The importance of customer collaboration requires the team to build an atmosphere of mutual trust and open communication. It can be difficult to get some customers to participate actively in many of the processes.
Finally, there are challenges in moving away from a process-centric organizational system:
- Several organizations have invested heavily in CMM practices to strengthen processes and ensure stability in their setup. Moving to Agile requires them to look again at these practices and do away with rigid processes that hinder teams from working well using the new methods.
- Traditional life cycle processes are elaborately defined, measured, and controlled to minimize variations. In Agile, by contrast, project management has to rely on assessment rather than measurement and has to keep the processes highly customizable.
- The practice of writing code and then using test cases is entrenched in traditional methods. This is reversed in Agile practices, where test cases are written before any code is. This redefines the function of quality assurance and control departments in the organization.
The successful transition of organizations to Agile methods depends on careful consideration of organizational, personnel, and process aspects that need to be changed. Organizations may need to consider a combination of traditional and Agile methods, based on the project's size, complexity, customer readiness, and need for speed.
Augustine S. Agile project management: Steering from the edges. Communications of the ACM. 2005; 48(12):85-89.
Nerur S, Mahapatra R, Mangalraj G. Challenges of migrating to Agile methods. Communications of the ACM. 2005; 48(5):72-78.
Takeuchi H, Nonaka I. The new new product development game. Harvard Business Review. Jan-Feb 1986.