A large system-integration project, on its own, is a challenge for a development team and its stakeholders. Having to adopt the Agile method to implement a large-system integration project can be an even more challenging situation if the development team is not empowered by the organization, or if they are not familiar with Agile.
What is clear is that if an Agile team is not allowed to take ownership, it cannot be accountable. And if it is not accountable, it will only be committed to tasks that make it evident that it is a team lacking collaboration and self-organization.
To succeed with a big integration project, the Agile team must be empowered. The bigger a project is, the more likely it is to be integration dependent. And as a project becomes dependent on integrating modules, the capacity for development work will begin to shrink; yes, this is true. Big project development tasks tend to decrease in order for infrastructure works to increase because of the amount of integration work required. It will be rare to find a big project with a small amount of integration work.
The challenge, however, is that there is strong tendency for a project team to put a lot of effort into coordinating rather than collaborating and self-managing. This could also lead to a project manager or ScrumMaster inadvertently managing directly rather than allowing the Agile team to be self-organizing and self-managing.
When an Agile team is hindered from being high performing and self-managing, the typical shortfall is that organizations will notice a sharp increase in their overhead cost of the project, part of which could be the costs of directly coordinating the team and hiring. I have been at a client site where ScrumMasters and technical leads were hired to facilitate product development and then report to a project manager. The project manager coordinated the project and supported the team for success. This approach is not the best when managing a large integration project; it has its own peculiar shortcomings.
I want to unequivocally state that highly demanding, complex, and large integration projects more often than not require a project manager and ScrumMasters, but the project managers should operate at the strategic and not at the team-delivery level. The project manager should facilitate and represent the interest of the team at the strategic level where some level of political skill is required.
On the other hand, Agile technical practices or principles may be impeded by an organization's long-time culture, people’s mindset, values, or even a shroud governance. I call them determinants
. The success of an Agile project anywhere can hinge on any of these determinants. For many organizations, these determinants have directly or indirectly become the impediments in the way of implementing Agile.
To implement a large system-integration Agile project, with all of its inherent complexities on one hand and impeding determinants, such as the organization's long-time culture, people’s mindset, values, or even a shroud governance, on the other hand, is tantamount to crashing even before takeoff.
Let me be clear that neither the inherent complexity of a large system integration nor the organization's self-made hindering determinants need alone make a project fail. Rather, it is the lack of simplicity. If you don’t keep the project simple and stupid, you will inadvertently complicate it. It might not be what you planned, but it will happen.
The challenges of a large system integration in Agile can be made worse when an organization expects some of the deliverables from external developers (vendors). I worked for a client that had trouble getting one of its vendors to fulfill their service-level agreement (SLA). The vendor would not be part of the Scrum meetings, and they could not deliver modules on time or stabilize data in their integrating environments for my client to consume. The project boat was rocked, and the team’s high spirit dissipated. My client could not do much, because the API call from the vendor's component was integral to the overall solution. I realized from this particular experience how the concept of a successful distributed Agile development team could be artificially made difficult to accomplish. Even though the project had a senior project manager and an experienced technical lead, there was still little they could achieve because of the constraints inflicted by the vendor.
For complex, large Agile system-integration projects, there is no template for success, but there is a recipe for working software to be simple, open-minded, and collaborative.
There is no template for success, so the way to go for working software in a large integration project is to be open-minded about continuous improvement. If the team, product sponsors, stakeholders, vendors, and individuals involved in a large integration project are not open-minded and approach everything with simplicity, they will have to contend with the mindset of complex architecture, complex test and development strategies, and people who think things will only get worse before they get better.
Agile project stakeholders who are not open-minded will inadvertently equate their current project to a previous one of similar propensity that struggled or failed.
When a team is partially colocated, it is highly likely that some of them will form the opinion that distributed Agile development teams can never be as productive as a colocated team. It will become easy for them to also assume that Agile works better with small teams.
A project manager in an environment where there are all sorts of Agile misconceptions will become defensive in his or her approach and probably begin to run a mini Waterfall in an Agile environment. The project manager will spend more time preparing to manage stakeholders’ expectations than treating this current project differently and approaching it with an open mind. Being open-minded and simple means that the team or individual team member will tolerate change and be flexible, and being flexible illustrates Agile in terms of attitude. A team or team member who is flexible is usually determined to make things work for the delivery of working software. Also, being rigid about the Agile technical practices or principles will only make things worse, as it could lead to being prescriptive and mechanical about Agile adoption and implementation.
The Agile team should be open to explore, able to adapt to change, and collectively choose whatever Agile technical practices or principles they are comfortable with as a team; otherwise, those same Agile principles might become impediments to its success.
Multiple product owners
Let’s address issues that might be misconstrued while using Agile software development methods to implement a large system-integration project.
Insisting that the business nominate one product owner in a complex system-integration project ignores the reality that no one single person in today’s complex organizations can have answers to everything.
From practical experience, the complexities of the system integration, the dependencies, and different stakeholders’ expectations could make having one product owner both unrealistic and unworkable. Also, from my experience, multiple product (component) owners may need to be represented in any major decision that impacts their domain. Often a business analyst with good domain knowledge might stand in where it is practically impossible for a product owner to be available to a team. While there could be a single product owner for the final product, the ideal thing to do is to get a single product owner for each component that integrates into the final product/solution so that timeline and goals can be set for a realistic delivery.
The idea of multiple product owners will work well as long as there is an agreed-upon process prioritizing the backlog. This does not stop the product owners from performing other roles that are traditionally performed by a single product owner. Scrum of Scrums is also an Agile practice that may be applied in this case.
It’s very clear that there are a couple of things within and outside an Agile environment that could impact the planning and implementation of a large system-integration Agile project.
Agile principle: "Working software is the primary measure of progress."
This Agile principle can easily be translated to mean being able to selectively adopt, adapt, and apply whatever Agile practices help you reach the finish line and deliver projects successfully. With an open mind, it is possible. If a team is empowered and allowed to be self-organizing and self-managing, they can select whatever combination of Agile frameworks, principles, and technical practices they are comfortable with to achieve the goal of working software.
Simply tailor your Agile to scale in a large system-integration project. This is different from partially implementing Agile; this strategy will not prevent an organization from maximizing/optimizing the full benefits and potential of the Agile method. To tailor in this context means allowing the development team to decide what works for them so that they can commit to delivering the goals.
Agree on segmenting the deliverables so that no component or delivery is delayed because of its dependency on another component that is under development or has yet to be delivered. Get the infrastructure/component team to run a sprint ahead of the features team so that the features team does not run out of work or is denied the use of a key component.