The Burden of True Agility

13 September 2013


Introduction

What do we know from a quarter-century of software projects? Most of them are either challenged or fail. They run over budget, run late, run off the rails. They cost more, take longer, and deliver less functionality than they should. The bigger a software project is, the more likely it is to fail. The bigger it is, the more likely it is to have all the trappings of traditional software development: the extensive plans, complex negotiations, detailed documentation, and painstakingly defined processes.

 image-1-(1).jpg
Article source: managingagile.blogspot.com
 
Agile suggests that the grass is truly greener on the other side. However, many project management practitioners seem to stumble over reality on their way to the pot at the end of the Agile rainbow. To lessen the pain of that fall, here are a couple of common myths that organizations must overcome in order to extract the most value out of their attempts to speed up the delivery of software projects using Agile approaches.
 
  • Established organizations often claim that Agile does not work because it insists upon utopian qualities such as self-direction, self-governance, and self-management, which are most likely the qualities seen in a start-up venture but that seldom exist in the workforce of an established organization.
  • The flexibility of Agile has become the bane of its effectiveness for software delivery, because it avoids prescribing and relies more on guidelines and recommendations that are often abused and end up creating chaos.
Before addressing the above myths, let us take a moment and think about the lowest common denominator among all of them. This goes back to the first principle of the Agile Manifesto: "Individuals and interactions over processes and tools." True agility requires proactivity from team members that contributes actively to creating solutions, avoiding a reactive attitude toward challenges and changes. It is imperative for organizations to understand that Agile does not solve problems but helps them surface. Further, it offers a suite of methods and best practices that can enhance existing processes by embracing change while balancing flexibility with stability via specific controls.
 
This article is intended to focus on analyzing the gap that exists between the theory of Agile, its usage for software delivery, the trade-offs that are being made to realize success in both the short and long term, and its potential backlash, that inherently govern the success and failure of the process as a whole. The next few sections will elaborate a coach's findings and insights about certain types of project- and organization-specific challenges and how one can leverage pragmatic Agile techniques, whose flavors might stray from the textbook definition of the same, to overcome these challenges and consequently structure a robust process paradigm across the enterprise for multiple organizations.
 

Predictability and scalability

One of our recent coaching engagements was with the Web development division of a Fortune 500 retail firm that attempted to imbibe Agile into its software delivery process to expedite time to market of its products. An IT transformation group (consisting of IT leaders/program managers and method and best-practices trainers) was organized to strategize and shape the specific flavor of Agile that would fit into the organization's dynamic. After numerous training sessions, about a dozen project teams were formed and tasked with implementing various features for the company's website to facilitate online shopping and product delivery.
 
After a few successful rapid-value deployments, the company started to experience delays in release schedules and observed an overall downward trend. A few teams experienced a crunch in resource bandwidth. This was primarily attributed to the matrix nature of the organization. Another factor that contributed to the delay in release schedules was sharing of project infrastructure across multiple teams (e.g., hardware/software/environments, etc.), and this would often create chaos during the execution phase of projects. Despite the repetitive nature of these issues, the organization never addressed the root cause and simply attributed it to process failure and certified that Agile neither guaranteed predictability of success nor did it provide early warnings of failure.
 
Achieving predictability requires a program of intentional improvement at the team, program, and portfolio levels, supplemented with needs-based training, demonstrating results via Agile health metrics and progress via assessment scores. Most organizations also take assistance from an enterprise-level Agile coach who will help the teams be as predictable as possible given their constraints: one who can help produce factual information about the velocity/cycle time and quality that teams are able to achieve, and who can credibly express the impact of organizational behaviors that are reducing the company's potential (the PMO at the retail firm mentioned above chose to ignore this option due to budgetary constraints). Getting predictable in an enterprise is not a quick process, and an enterprise-level Agile coach can serve as a guide, interpreting assessments and metrics, relating training to day-to-day events, and cementing lessons learned in training sessions and workshops.
 
image-2.png
Source: managingagile.blogspot.com

A recent addition to the Agile framework, the Scaled Agile Framework for enterprise (SAFe) provides extensive guidance and practices for implementing software agility at enterprise scale. This model of Agile adoption serves well in an environment that comprises several cross-functional teams that are heavily interdependent and deploying in short and regular bursts. The benefits of implementing such a framework in the scenario mentioned above would be threefold:
  1. The Agile Release Train (ART, which is to a program what an iteration is to a team) would ensure a reliable release schedule at the enterprise scale, because if an Agile team wanted its "cargo" (code/documentation, etc.) to go, it had to put it on the train on time. The train would not come to the team or wait to get its content.
  2. Impediments common across releases would be exposed early.
  3. Resource planning and allocation would also be easy, due to preset assignment of resources based on priority/position of the project's release schedule on the ART.
Hence, when multiple teams work within a single process framework, there is a need to outline a consistent approach to release planning that ties together a consistent set of deliverables from multiple teams to deliver value at the enterprise level. Emphasis should be laid on establishing a regular cadence of planning, execution, and delivery that all teams follow within the enterprise. Agility is about adapting to change and letting processes emerge, but it also calls for maintaining alignment with the vision and strategy from top to bottom across the organization to ensure a continuous flow in the product development lifecycle.

Command and control and empirical process

Many organizations perceive "self-organized" or "self-managed" teams as teams that lack direction and focus and hence cannot generate effective results. On the contrary, this is a misunderstanding. A couple of years back we coached the IT department of a Fortune 500 financial firm. This company wanted to radicalize its development process by cutting away from the "all or nothing" approach to an iterative method and, in time, foster an overall Agile framework in the organization. As initial steps, an Enterprise Transformation Team was set up comprising of Agile coaches, trainers, technology writers, and methodology experts.
 
Initially, everyone at the organization was excited. However, after a few months they reached the conclusion that teams can be Agile but a formal project manager is required to ensure acceptable results and feed the need for regular status reporting, PowerPoint decks, and comprehensive project report metrics for senior leadership. The symptom in this case was fear of losing control and lack of understanding of a "self-organized" team. Instead of realizing the root cause, the company chose to address the symptom by adding a point person responsible for "if all fails." This "if all fails" thinking drives fear of losing control and often curbs flexibility, independence, and creativity.
 
The aforementioned strategy seldom works; especially in keeping abreast with today's rapidly changing technology and competitive edge, one has to inspect and adapt as and when the opportunity presents itself, rather than following a prescribed plan. The failure to let go of command and control is usually due to result-oriented thinking. The idea may work in a predictable environment where consistent input and process steps yield a determinable output, but in today's world, we deal with complex adaptive systems where the results are heavily based on empirical processes. Agile is a big proponent of empirical process-oriented thinking, which promotes a focus on improving the process (P criteria) before getting an improved result (R criteria). Hence, before a desired result is achieved, the focus should be on how to improve and optimize the process to achieve stability and sustenance.

image-3-(1).jpg
Source: Kaizen: The Key to Japan's Competitive Success, Masaaki Imai
 
A well-documented plan is no good if it ignores regular feedback loops to readjust to new challenges and opportunities. Traditionally, this is executed as a "lessons learned" or "postmortem session," which usually happens toward the close of a project when it is too late to change anything for the current project. The knowledge gained can only be applied to another project, which would have its own unique challenges. As humans we are resistant to change, as change is unknown, often unpredictable, and may not have a defined line of sight. We need a process that is derived from experience, embraces change, and adapts to it.
 
image-4a.png
Source: leadinganswers.typepad.com
 
Agile calls for "Responding to change over following a plan." How do we achieve this? Well, we regularly assess, observe, and react to the changes by inspecting and adapting to reality. It is imperative to review the process and the final work product at regular intervals to assess the need for improvements and process changes while there is still time to make a difference. The whole idea is to continuously improve or change for the better (Kaizen).
 

Conclusion

Creating visibility from the top down and the bottom up and gaining better insight into planning, development, and integration is all an evolutionary process. Persistence, consistency, and solid daily reporting of progress are the key to successful implementation of Agile practices and methods. Timely and effective retrospectives can be another catalyst that helps teams understand the process better and affords them the willpower to press on. Agile is not merely a process for coding, it is also about how you educate your team members to have team spirit, be open-minded for discussions, accept other opinions, and be ready to help other team members. In essence, it is simply a way of doing things that has grown organically out of the need for feedback and communication.
image-5.png 
Source - keeping-agile.com
 
It is important to understand that no two teams are alike, so it is impossible to outline a full and robust process that will work for everyone in every situation. The goal should be to stick to the key tenets, but don't be afraid to fill in the blanks on your own. Just make sure to do so with an Agile mind-set. It is also important to recognize that Agile can be painful. It is designed to make impediments visible, and that means hard conversations. As much as we would like to pretend otherwise, we're all only human. Our bosses are human. Our coworkers are human. Our clients are human. We make mistakes and sometimes we don't get along. Conflict is natural when dealing with different people and personalities. Let it happen and learn from it. Inspect and adapt.
 
It can be too easy sometimes to focus on what needs to change to get better rather than what can be improved to get better. Just because a team excels at something doesn't mean that they can't push it further! Good teams always find ways for even the most refined process to be improved, and a successful Agile team is one that constantly evolves by never entering the comfort zone that's brought on by sticking to apparent tried-and-tested methods.
 
In conclusion, to be truly successful, a project team should continue doing the things that go well, stop doing the things that don't, and continue to find new ways to work that will ultimately make the team more efficient. Without wanting to belabor the point, Agile fundamentally works because of the consistency and routine that is built into each sprint/iteration/cycle.
 
Acknowledgments

Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.



Article Rating

Current rating: 5 (4 ratings)

Comments

Zach Bonaker, CSP,CSM,CSPO, 9/13/2013 10:01:39 AM
Well organized and nicely sourced. Thanks for contributing this article; I found it a worthwhile read!
Raju K, CSP,CSM, 9/15/2013 9:32:28 AM
Good article with relevant graphics. My CSM journey started today and your article answered few of my questions.

You must Login or Signup to comment.