“If you can’t measure it, you can’t improve it.” That quote is, of course, attributed to Peter Drucker, management consultant, educator, and author. A company’s transition to Agile is a continuous journey that involves cultural change and adaptation. As a company assesses its progress toward becoming Agile, it needs a well-understood and shared metric for its success.
Evaluating our Agile progress across the enterprise is critical, but it is not the ultimate measure of our success. In 2014, Bottomline Technologies published an article, “Agile and the Organizational Ecosystem
.” In that article, we included a quote from our president and CEO, Rob Eberle, in which he outlined the metrics we measure as our drivers of success.
“The future of our business is dependent on our continued ability to bring innovative, high-quality, and high-value solutions to market ahead of the competition. This is an extremely important initiative for our company — Agile represents an opportunity for us to improve all of our key metrics: Innovation, Feature Function, Quality, Time to Market, Cost, and Growth.”
So began the proactive “Growing” phase of our enterprise-wide journey to becoming a more Agile organization. Prior to 2014, we had a few teams using aspects of Scrum, but they were disparate and not part of a larger, enterprise-wide community.
From 2016 to 2017 we saw continued progress toward our Agile goals, with growth in the number of Scrum Teams from less than a handful in 2012 to more than 60 Scrum (and Kanban) teams in 2016. All of our new product development teams are now Agile. Below is the roadmap of our Agile journey, from our “Early Agile” years in 2012–2013 to our “Growing Agile” years in 2014–2015 and our “Maturing Agile” years in 2016–2017.
A key element of our Agile maturity is the ability to measure progress at the team, organizational, and enterprise levels. As anyone who has ever undertaken the effort might attest, this is no small task! Following Agile principles, we started early and small in scope, then iterated and grew the process as we learned what worked and what we wanted to accomplish. As seen in our roadmap, Agile maturity measurement is one of our targeted goals in 2016–2017.
Bottomline technologies and our Agile publications
Bottomline provides cloud-based solutions to major institutions and corporations around the world. The company's solutions are used to streamline, automate, and manage processes involving payments, invoicing, global cash management, supply chain finance, transactional documents, legal spend management, healthcare information, and cyberfraud and risk management.
We have published several articles about Bottomline’s Agile journey, including:
There have also been two case studies highlighted by Scrum Alliance®
The Organizational Ecosystem Model
At the core of our Agile maturity measurement is our Organizational Ecosystem Framework. This model was introduced in our first publication listed above, “Agile and the Organizational Ecosystem.” Below is an overview of the Ecosystem Framework Model to provide context for understanding our Agile maturity measurement approach.
The premise for introducing the model was the belief that an Agile transition that is not aligned with or that does not support organizational needs is not sustainable. The article identified four Agile change components for teams to assess:
- Organizational Dynamics
- Product Management
- Technical Management
- Agile Principles and Scrum Practices
As teams review the elements in each of the four components, they move from the grey, unassessed area in the framework to the Green/Yellow/Red areas, indicating which elements support Agile principles and which need to become Agile strengths. From this assessment, teams can create a priority action list and lay out their transition plan.
We have used this Organizational Ecosystem Model successfully at Bottomline with multiple Agile transformations, with very positive results. We have seen velocity increases that have ranged from 250% to 500%, along with quality improvements, in just two to three release cycles! Details on how to apply the Organizational Ecosystem Model are included in the article link provided earlier.
Why measure enterprise Agile maturity?
We recognized that a successful transformation was only the beginning of the journey. We knew we had to have a way to measure and monitor our progress as existing teams and products evolved and as new teams were formed. Sustainability is a primary objective for healthy Agile organizations!
In collaboration with Jeff Sutherland, we adopted the Nokia Test. The Nokia Test for Scrum teams was developed originally by Bas Vodde at Nokia Siemens Networks in Finland. It has been updated several times and appears in its latest incarnation in Jeff Sutherland's Scrum Certification classes. The test includes ten assessments that cover the Scrum process, events, roles, and artifacts. The Nokia Test’s stated objective is, “As an Agile citizen, I can assess a team’s behavior and compare it to current Scrum best practices, so I can consider changes that might increase productivity.”
The Nokia Test was a great way for us as an enterprise to get our teams to think about how well they apply Scrum and what steps they need to take to strengthen their use of Scrum. As we began to look at our organizational Agile maturity, we knew we had to go beyond individual Scrum Team measures and expand our view to the broader measures of our successful Agile transformations.
As part of our continuous improvement, we looked to our Organizational Ecosystem Model, which focused on these broader measures. We incorporated the four Foundational Components into our Agile maturity measurements. By expanding our approach to include these factors, we raised the expectations for not only our teams but also for our product organizations and our entire enterprise.
Agile maturity benefits
Measuring enterprise Agile maturity is important for many reasons. These include the benefit of knowing how well each part of your organization is maintaining its Agile culture and assessing how business performance is doing in light of the Agile maturity results. Measurement also enables more effective resource allocation decisions with limited Agile coaching and training resources.
Other benefits of measuring our enterprise Agile maturity include:
- Transparent, consistent, and continuous focus on Agility as a priority across the enterprise
- Regular gap analyses, effectively incorporated as part of a team’s release retrospectives
- Meaningful appreciation for each team’s efforts as well as increased stakeholder support
- Linking performance benefits and potential gaps to Agile maturity–level improvements
- In addition to these efforts, we are building Communities of Practice (CoPs), within and across our organizational areas, to facilitate learning and knowledge sharing. These CoPs align with the roles of product owners, ScrumMasters, developers, quality assurance, and Agile managers (managers of Agile resources), with a focus on the priorities and challenges that are important to each group.
As part of this effort, we have developed “Next Level” Agile Workshops with a focus on these roles and strengthening the skills to achieve organizational success. These workshops highlight the CoPs as a key initiative to our enterprise success. It is our expectation that we will shorten our learning curve and reduce the time needed to achieve Agile maturity with each new team added to our organization by creating and enabling this support structure within the enterprise.
Measuring Agile maturity
Having discussed the importance of measuring Agile maturity, this section introduces the thought process behind creating the Agile Maturity Model, along with the mechanics of the measurements.
Organizational Ecosystem Framework
Agile teams do not operate in isolation but are part of the larger organization. Hence, to improve overall effectiveness and drive process efficiencies, the efforts of teams need to be complemented with the organizational support of upstream and downstream partners.
This perspective is congruent with the findings of Version One’s 10th State of Agile survey,1
which suggests that the leading cause of failure in Agile projects is company culture being at odds with Agile principles and practices.
Some other leading causes of failure are a lack of management support, ineffective communication and collaboration, and a lack of experience consistency in following Agile practices. Several maturity measurement frameworks focus deeply on improving the “techniques” side of practicing Agile. We believe it is vital to consider the holistic environment in which the team operates, along with the Agile principles and practices, for an effective Agile maturity measurement framework.
Some additional questions may include, “How would you know which elements of the overall environment or ecosystem need to be in place?” and “How would you know which elements are supportive or need more traction?” To answer these burning questions, the Agile Maturity Matrix has been designed to encapsulate these Organizational Ecosystem elements.
Like the Organizational Ecosystem, the Agile Maturity Model consists of four major areas crucial to achieving and sustaining Agile maturity. These are noted below:
“Agile Principles and Scrum Practices” concentrate on the doing
part in improving Agile expertise, while the other three sections, namely “Organizational Dynamics,” “Product Management,” and “Technical Management” concentrate on the enabling
part in supporting teams on their continuing journey to Agile maturity.
As such, the Agile Maturity Model arguably offers a holistic improvement mechanism; however, these four areas are quite broad. The key challenge is distilling them effectively to identify the crucial aspects to measure. This distillation challenge led to us defining five key elements within each of the four ecosystem areas, as shown in Figure 4.
To ensure that the understanding of these elements is calibrated across the organization, a brief definition of each element is provided. This allows teams to understand the context more easily.
For example, when analyzing the element “Business and Technical Relationship” under Organizational Dynamics
, it helps to clarify what needs to be focused on. Specifically, we define the element “Business and Technical Relationship” as “Collaborative participation by stakeholders is encouraged” and “Development is not seen as a technical activity in isolation.” Setting context clearly is important to help ensure that different teams can share the same understanding when evaluating their maturity. It also helps to maintain a shared understanding and context on each team over time, facilitating a more consistent and targeted improvement measure.
Scoring mechanism and results
For scoring in the Agile Maturity Model, we used the Japanese martial arts stages, from learning
, known as Shu Ha Ri
. The concept is shown in Figure 5.
Among others, Alistair Cockburn2
has written quite extensively about applying Shu Ha Ri in Agile maturity measurements to represent the stages of learning through maturity. A team’s journey starts with Beginner
(Shu state), then progresses to Intermediate
(Ha state) and Expert
Corresponding to the stages of maturity, the Shu, Ha, and Ri levels have each been given a score value of 1, 2, and 3 respectively to quantify an overall Agile maturity score for the team. Each of the elements of the Agile Maturity Model as mentioned in section 3.2 has a corresponding descriptor that specifies what attributes qualify for Shu, Ha, or Ri level states of mastery. Providing these unique descriptors helps teams with a more consistent understanding of the qualifying criteria for each level.
Scoring example — Team level
For instance, Figure 6 below shows the Organizational Dynamics section with two of its attributes and their descriptions. These are followed by Shu, Ha, and Ri–level outlines for each element. A score of 1, 2, or 3 is then noted for each attribute of the team undergoing its Agile maturity measurement.
Once the team has scored all 20 elements of the Maturity Model (4 Ecosystem Areas * 5 elements), an overall Agile maturity result for the team, ranging from 1 to 3, is calculated. The results of the Agile maturity assessment are displayed in the form of a dashboard that is comprised of four spider charts. Each spider chart corresponds to the four areas of the Enterprise Ecosystem, as shown in Figure 7.
This dashboard provides an at-a-glance overview of the Agile maturity status in all of the areas targeted by the Agile Maturity Model. This view helps both the team and stakeholders visualize areas that are maturing well, along with those that need more attention.
The dashboard below for a sample team suggests that, within the “Product Management” arena, the team is doing very well with “Product Vision & Incremental Value” (reported as 3, or Ri state), which indicates they are more mature. By comparison, “New Feature Estimation” and “Product SMEs” (both reported as 1, or Shu state) indicate less maturity. The dashboard is a powerful diagnostic tool to analyze a team’s overall maturity and its corresponding strengths and targeted improvement areas.
Application in Bottomline Technologies
Teams across Bottomline’s suite of products have incorporated the Agile maturity assessments as part of their continuous improvement measures.
The simplicity of the Agile maturity assessment tool has been a key driver for teams in accelerating its adoption. Feedback, as captured in Figure 8, suggests that the assessment tool, while being straightforward to use, is very effective in fostering meaningful conversations about its various maturity elements.
These rich conversations among team members are the essence of conducting the Agile maturity assessments. To sustain a successful Agile environment, the organization must have regular discussions regarding its evolving situation. For these discussions to be productive, people’s assumptions about the organization’s capabilities and its interactions must be honest and transparent. In addition, the drive for change must be shared and supported by management.
To ensure continued measurements and improvements over time, we coached teams to assess their Agile maturity at release-end or quarter-end, whichever works better for them. The time frame arguably represents just enough opportunity for concentrated improvements. It is long enough to follow through on the actions to fill the maturity gaps, while being short enough to keep the team focused and concentrated on continuous improvement.
Another aspect we coached was to include the measurement of Agile maturity as part of the Release Retrospective agenda. The outcome of the assessment also offers a good opportunity for calibrating the wider organization around the Scrum team. This cadence, if maintained for each release, aligns the healthy interactions necessary for long-term Agile maturity.
Scoring Example — Product Organization and Enterprise Levels
The result of the assessment for a single team is one overall score, along with the summary dashboard for that team, as shown in Figure 7 in section 3 above. The overall score of multiple teams delivering a single product can be used to have a good view of Agile maturity across the product organization, as shown in Figure 9 below.
As one can see, this representation offers a good at-a-glance view of the Agile maturity for each product organization within the overall enterprise. It offers great visibility and transparency and can drive effective communication among teams, stakeholders, and executives.
Finally, the maturity measurement system scales to analyze Agile maturity from the product organization
level to the enterprise
level. Using the overall scores for the individual product organizations, the enterprise level maturity scores can be derived.
Figure 10 below shows a graphical representation of the product organization scores within the context of the enterprise. Each product organization score results from the aggregated team level scores represented in Figure 9 above.
This view can enable effective decision making, based on empirical evidence, within and across the enterprise’s portfolio of products. When the leaders of the product and development organizations get together, they can see quickly how well their organizations are doing and where to improve.
When presented as part of each organization’s overall progress against its business goals, the Agile maturity assessment provides an additional and valuable tool in strategic decision making, from measuring drivers of success or struggle to targeting improvement opportunities. In addition, where enterprise Agile Coaches are limited, this assessment can be useful in deploying these resources for greatest effect.
At the start of 2017, we find ourselves actively in the “Maturing” phase of our Agile Roadmap. This is an ongoing journey, for the organization's Agile maturity and the evolution of the model itself.
In addition to the Agile maturity measurements for all of our Scrum Teams, we are aligning our organization to more effectively incorporate a DevOps approach into our end-to-end product delivery. This approach enhances our ability to delight customers by delivering feature-rich releases more efficiently, with higher quality, minimal post-production lag, and enhanced support.
As we continue to mature, we will evaluate the potential benefits of evolving from Scrum Teams to feature teams (cross-functional, cross-component, end-to-end) that deliver customer features. While maintaining team stability is an important goal in an Agile organization, an additional benefit of “growing” and “maturing” is that we can leverage team members (or entire Scrum Teams) cross-functionally across teams or organizations with less downtime and greater effectiveness. All of these enhancements can potentially be incorporated into future versions of the Agile Maturity Model.
As highlighted by our Ecosystem Model, technical and product management practices throughout the organization are critical to our overall enterprise Agile maturity. All of our teams are tracking their levels of current test automation and driving quarterly improvement targets. Our teams are also advancing their continuous integration and deployment automation capabilities.
In addition, most product release schedules have been reduced from annual (or longer) to quarterly. Product management teams continue to review and assess with customers the needs and benefits for more frequent release schedules in their markets. Product teams are leveraging standard KPIs, focused on critical business measures, as part of our transparent and effective reporting efforts. Toward that end, we are implementing the Jira Portfolio to enhance our Atlassian suite capabilities. Achieving enterprise Agile maturity is a collective effort. Bottomline is globally distributed with a diverse set of products, Scrum teams, and cultures. While we have achieved 100% adoption in the use of Agile and Scrum for all new product development efforts across the enterprise, we have to continue to work together to grow and sustain our Agile maturity.
A key element of this collective effort is an enterprise-wide Agile/Scrum wiki to provide a common place for people to share tools, techniques, articles, books, and more. We maintain this wiki in Confluence and have expanded it to house the Agile maturity results for our Scrum Teams. Figure 11 below is an image of the template we have created to capture each team’s Agile maturity results and the identified actions to strengthen their maturity before their next measurement checkpoint.
Team Agile maturity results are then rolled up to the Product Organization level and, ultimately, the Enterprise Level, as shown on the Confluence page below in Figure 12.
As mentioned previously, feedback from the teams on using this model for measuring Agile maturity has been positive, commenting on how straightforward it is to use and how effective it has been in fostering meaningful conversations.
Similarly, feedback from organizational leaders and managers across the company reflects their perspectives on the gains to be made by keeping a consistent vigil on our Agile maturity to ensure sustainable results. Several leaders in our organization shared these thoughts:
- Andrew Mintzer, EVP Product Strategy & Customer Delivery: “Agile provides the transparency to measure ROI and helps us make smarter business decisions. With multiple, concurrent projects occurring across our enterprise, our Agile maturity is a critical component to delighting our customers and achieving our business and financial objectives.”
- Avital Serfaty, Product Manager for Cyber Fraud and Risk Management: “Agile maturity measurements, when conducted regularly as part of the release retrospective, offer us a great vehicle for effective gap analysis. Teams share their experiences in different areas of measurement and define kaizen (improvement actions) to achieve greater Agile maturity and more effective delivery. This has also helped our product managers and Scrum Teams to collaborate more effectively on a shared vision to develop ever-compelling products for our customers. Teams actively utilize the Agile maturity measurements as a means of continuous improvement, enabling them to make a meaningful difference.”
- Jason Cohen, Director of Development for Paymode-X: “Strong stakeholder support of a team’s consistent delivery of value and quality leads to a growing partnership between the business and technology. This acknowledgment of the investment in each other’s success and focus on continuous improvement results in a seamless appreciation of team efforts.”
- Scot Calitri, Senior Director Program Management for Legal Spend Management: “Consistent measurement has allowed our business to capitalize on team-identified opportunities (Shu or Ha items) while sharing patterns and techniques associated with our stronger (Ri) areas of measurement. Having the entire team participate in regular assessments helps live the peer relationship that Agile principles are founded upon.”
By partnering with product organizations and teams across the enterprise, we are making progress toward our Agile maturity goals. It is not a linear pattern of progress, to be sure. Different product organizations and teams face different challenges and progress (and sometimes regress) differently.
By always keeping a priority focus on delivering customer delight and having a shared measurement approach across the enterprise, we can more effectively identify our strengths and target the appropriate product organization and team-specific actions for improvement.
1. 10th Annual State of Agile Report. Version One, http://stateofagile.versionone.com/
2. Shu Ha Ri, excerpts from Agile Software Development, http://alistair.cockburn.us/Shu+Ha+Ri