I recently completed a long assignment implementing Scrum and decided that, in the spirit of continuous development and learning, it would be useful to run a personal retrospective while between jobs.
This paper articulates some of my key findings and probably confirms what many Agile practitioners already suspect. I have suggested some techniques and ideas that may be effective in managing future transitions to Agile. I assume the audience for this article is already familiar with Agile methodologies, so I'm not going to repeat them.
On the surface it would appear quite easy to transition an organization to Agile, but you shouldn't underestimate the strength of resistance you may encounter. Not many people have worked in a truly Agile environment, so there may be a fear of the unknown or "how it's done," and a compulsion to stick with what they know — i.e., Waterfall.
On the whole I found that the developer community seemed to be more outward looking and receptive to change. Most bear the scars of previous projects and are willing to give Agile a go.
People working in supporting functions or the broader business can have strong beliefs about "the way we do things around here." Even when there's clear evidence that their assumptions are incorrect, they will still make any number of excuses to justify them. If you're only measured against your last project, there's terrific pressure to claim success, even if the project delivered very little value.
I came across some organizations that claimed they had tried using Agile but it didn't work for them. When taken to task, it was clear they had made no real effort to transition to Agile, preferring instead to maintain the status quo. Their project managers kept running projects but called them Agile projects. They retained their business analysts and technical architects, so the results should hardly surprise any of us.
It's easy to drown in detail, so now I have attempted to step back from the situation and apply a more systemic approach to the problem. Therefore some of the activities I propose here may seem counterintuitive. For more information about advanced problem solving, please refer to Stafford Beer's Viable Systems Model (VSM) and Peter Checkland's Soft Systems Methodology (SSM).
Here are some of the issues I encountered. I'm sure you're also familiar with many of them:
- Wanted: Agile project manager. Spot the warning signs early on. An organization looking for Agile project managers is probably running projects and may have heard that Agile is the way to "do more for less" — i.e., save money. The role here will need clarification. Why does the organization want to transition to Agile? Is it to save money, do more for less cost, or do they want to keep running projects and create the illusion of Agile?
- The governance excuse. We can't govern Agile projects like Waterfall projects, where there are the gates and decision points. How can we control the budget, where is the stakeholder engagement and accountability?
- Lack of leadership support. Without buy-in and support from the top, you're unlikely to succeed. Also watch out for middle managers pursuing their own agendas.
- Scrum as a process or method, not a guiding principle.
People don't always understand the principles underlying Agile methods. Scrum is a simple process based on these principles. The trouble starts when people start adapting Scrum methods without understanding the principles (like tinkering with your car engine without understanding how it works), which is why you end up with things like Water-Scrum-Fall.
- Project managers' beliefs. Watch out for project managers who believe it's a well-known fact that rigid adherence to a recognized project management method is the only way to deliver effective change, and if it's not working you need to do more "project stuff" (strengthen your change-control procedures and adhere to stricter governance).
- Agile is just another fad or new methodology. People believe they just have to ride it out until it goes away.
- The focus is on delivering projects instead of products and services. Is the focus of the organization on delivery, or have your projects become bigger and more important than the products they deliver?
- People don't know what Agile looks like. Can you share a vision of what a good Agile organization looks like? If somebody had never seen a sunrise, how would you describe it to them; what if you had never seen a sunrise either?
- External suppliers and offshore workers. These are usually based on fixed requirements and deliverables. Collaboration through contract negotiation and intermediaries isn't going deliver value in an Agile environment. Another approach is required to make these relationships work.
- A lack of trust or communication between the business and development communities. Agile principles are based on building a culture of trust and respect between both parties.
- Partial adoption, or Water-Scrum-Fall. The best you will achieve is mini-Waterfall; at worst you get suboptimal Waterfall.
- The use of estimates instead of story points. This is a subset of partial adoption. It places the focus on what things cost and not on what generates the most value.
- A big-bang culture. Storing up inventory and delivering it in a splurge of disruptive chaos gets noticed more than incremental change.
- Isn't the ScrumMaster just a project manager? Some organizations send their project managers to a Scrum course, expecting that to sort things out.
- The illusion of progress. Some organizations run lots of projects (even though they have insufficient resources), so people have to participate in multiple teams. They create lots of progress reports to paper over the cracks in the hope that nobody notices the true state of the project until it's too late.
- There is no "I" in "team." Some project teams are set up and disbanded so swiftly they never ever get past the storming phase.
Change and transition management
There's lots of information on organization structures and change. It was the focus of my MBA and it can end up being a lot more complex than many people imagine. That said, there are some tricks to herding these cats.
The move to Agile is very simple, but at the same time extremely difficult. Implementing Agile processes is the simple bit; stopping the doing and thinking in projects in order to deliver your products is the hard bit. It's hard because projects are probably embedded in your organizational culture, and to stop doing them challenges the values and beliefs of nearly everyone.
Every organization is different, and so their journeys to Agile will differ. I help people try to understand these five things:
- What the organization looks like. Is the organization large or small? Dynamic or inflexible? Focused on cost or value?
- Where the organization is on its journey to Agile. Just starting out? Some progress but now stalled or backsliding? Running Water-Scrum-Fall projects?
- The organization's ability to manage effective change. Is change embraced? Do people understand transition management? Are people more adept at resisting change than implementing it?
- The organizational structure. Function follows form. Does the form support its customers?
- Is the change to Agile being made within a subset of the organization? Meaningful change is likely to involve the broader business, not just a department or IS.
It's best to perform this analysis early on, while you can still smell the culture. Live too long beside a sewage farm and you won't smell a thing, and eventually you have to rely on indirect evidence. Why don't our friends visit us anymore?
Don't be surprised if simply employing a ScrumMaster doesn't deliver you Scrum. If you want to transition to Agile or Scrum, look for an Agile practitioner or a ScrumMaster with change-management skills. Don't assume a ScrumMaster can achieve the transition for you!
Sharing the vision
The transition to Agile is likely to require significant organizational change. Analysts may find themselves working for the business and becoming product owners. You may have to reduce your project management headcount. Not all change can be managed in a soft, comfy way, and sometimes a leap of faith is required. The change manager is there to support and coach the organization at all levels. Sometimes there's good news and sometimes it's less good; either way, it's important that everyone is involved and understands the need for change and how it's going to be achieved.
Not everyone has worked in an Agile environment. Big or small, it's important to include them in the journey. My favorite technique for creating a common understanding is through the use of a simple metaphor. I like to compare the Agile development team in an organization to a car production line.
In most circumstances it's probably best to run a project to build the new product (although this wouldn't be the case for a high-performing Agile team). Being pragmatic, something of this size probably needs funding and governance to get it moving in a large organization.
The question is, would you build a new production line and then tear it down every time you wanted to produce a new model? That's what you're doing when you run discrete development projects or Water-Scrum-Fall. Delivering Waterfall projects breaks one of the fundamental principles of Agile, by disbanding your team as soon as they have started to perform. In some cases the team never make it out of the forming stage, which is highly inefficient.
You wouldn't project-manage a production line, so why try to project-manage an Agile delivery system? Once the production line and product is up and running, using Agile or Scrum is the most effective way to maintain and improve the production line (Scrum team) and the product. Organizations running successful Agile don't run many projects (some run none), because their focus is on the product and delivering value.
Agile organizational structures
Part of sharing the vision is discussing what your organization looks like and how it needs to be structured. Function follows form, and this may need to change during the transition to Agile.
- Waterfall or Water-Scrum-Fall: The focus of the organization is on running projects. A functional structure relies on matrix management to build project teams. People's focus is on their function or profession.
- Transitioning: The focus of the organization is on delivering products through projects. It uses an Agile team structure, matrix managed by projects. People's focus is on the project and product.
- Agile: The focus of the organization is on delivering products and solutions. The organization is based on Agile teams. People's focus is on delivery to the customer.
Returning to the metaphor, the transitioning organization may have set up some production lines with projects running in parallel. The projects book capacity on the production line through a product-management function. The challenge for program management is to schedule the requirements across the production lines (this could be run like a Kanban system), keeping the teams fed at a steady rate (velocity). Any capacity gaps can be filled by the team with small change and maintenance (refactoring) tasks.
Why do organizations run projects?
Don't ignore projects. One of the first things I encountered at Agile forums was an attempt to ignore or treat project management as though it didn't exist. Living in an Agile world is as bad as living in project world!
I would argue that projects still have their place in the new order; creating the product in the first place is probably best run as a project. Once you've delivered the basic product, the most efficient way to change and improve the product is through Agile.
If you're transitioning to Agile, you need to take a hard look at the projects you run. Look at your proposed or in-flight projects and ask the following questions:
- Is this just a change to an existing product? Why run a project to implement a small change or improvement?
- Is this a big project? Is it really big, or has someone just bundled a load of stuff together to justify a business case, or made it big because small projects don't scale in a way necessary to support your big-bang culture? If it's just a lot of small stuff, why not challenge your Agile teams to break it down, remove the dependencies, and deliver it piecemeal. If they can't, you may have yourself a project.
- Will your customers accept iterative deliveries? Have a conversation with your customers and ask them whether the whole thing must be completed before they will accept the product. If not, push back and challenge them to answer why this is so.
- Will it deliver huge value? If you run a project, it's going to be high ceremony and incur significant overheads. Also you won't realize any benefit until the delivery. Make sure that's part of the business case. If the business case doesn't stand up, stop the project.
- Is the project delivering something that's required now, as opposed to something the customer actually wants when the project delivers it? How dynamic is your business environment? By the time you have your website delivered, will it be obsolete because your market and customers will have migrated to mobile devices?
- Is there little or no risk of change? Change is the enemy of the project and needs to be avoided. Managing change is built into and part of the Agile process.
- Does it need "project managing"? Once a project is delivered, project managers tend to hang around like a bad smell, convincing you that the only way to make change is through running another project. Keep checking: Do you really need to run a project to deliver the next change?
- Do you have a fixed budget? if so, Agile is probably the best approach, as you can stop when funds are exhausted.
- How do people know what's going on? Are managers content with reading project reports and reviewing risks and issues, or do they prefer to get out and see the progress for themselves (i.e., the Gemba)?
It takes time and effort to establish a project, but if you've just started to implement Scrum, I think it's best to stick with in-flight projects and accept them as a lost cause. As soon as the project team has delivered its product, you should be ready to establish your Scrum teams and take ownership of the product, run Scrum, and give control back to the product owner. The first few sprints may be needed to make the product maintainable and fit for the purpose, but after that the team can start to deliver some value to the product owner.
Implementing the change
I believe that everyone needs to experience the feelings of motivation and self-worth that being part of a high-performing team provides. It fosters respect, provides support, and really delivers. Scrum works because it recognizes that high-performing teams (not just a loose collection of individuals in a functional department) actually work. Self-organizing teams fulfill an intrinsic urge to belong and be much more than the sum of their parts. That alone is a good enough reason to transition to Agile.
To support the transition to Agile, ScrumMasters need to be more than simple servant leaders and facilitators. They have to be ambassadors of change. There is a lot of soft stuff surrounding change management, which really isn't an adequate approach to making the transition to Agile successful. In order to be successful, you have to convince people that it's safe to let go and take a leap of faith. It takes leadership and drive and it's not easy — but if it were easy, wouldn't we all be doing it already?
Change managers recognize that what you stop doing is as important as what you start doing. If you stop thinking in projects, it becomes easier to focus on products, value, and delivery.
Think about your organizational structure. Can you build and maintain a production line (or production lines) and support them so they can focus on delivery of your products? Set out and agree on some clear transition activities with measurable outcomes, then follow up on them. Alternatively, use Scrum to implement the change, listing each of your transitions as user stories and use sprints to deliver them. That's a great way to teach Scrum to the broader business.
Whatever your approach, it's important to keep up the pressure. If business managers want you to stop the transition, tell them to do what they agreed to. If you settle into a comfy compromise, you only have yourself to blame.
- It's nice to be part of a successful transition, but you may learn more from failing to implement Agile.
- If you only take away one thing from Agile, it should be the necessity to reflect and learn. Good or bad, reflect on how it went, learn from the experience, and remember: "The person who never made a mistake never made anything."
- If you only have project managers, the solution to every problem is probably a project!
- Are you content with generating status reports, managing risks and issues, and chairing meetings with angry stakeholders? Why not have a look at real governance, like Gemba — that is, get out there and find out what's really going on!
- If you can't stop projects completely, try to be prescriptive over the entry criteria. Make sure they scale and can't be done more efficiently by your Scrum team.
The move to Agile is very simple: Start doing Agile. It's also extremely hard: Stop doing projects.