Get certified - Transform your world of work today


Agile Pulse Checker

Taking Agile's pulse through audits

14 July 2015

Madhavi Ledalla

Agile has become the norm in today's software industry, mainly because it accelerates time to market, and it enables timely feedback that aids in building what the customer wants. The key aspect that makes Agile successful is the process framework that helps guide the execution flow.

We are now used to seeing instances where organizations start adopting Agile with full rigor. But after a few cycles, we see the process effectiveness taking a hit and organizations going back to the traditional way of working. Even if Agile is sustained for a long period, we see instances of nonadherence to process and Scrum-But kinds of symptoms creeping in, making the organizations wonder whether there was enough return on investment in transforming to Agile.

Consider the health-check metaphor. We go for periodic health checks to determine our health status, and we take corrective and preventive measures. Corrective measures include taking medicine for newly discovered ailments, and preventive measures include taking some precautionary measures, such as taking calcium and other multivitamins to prevent any future health problems. This periodic checkup is indeed required to ensure the health of the biological process of any human body.
Similarly to any industry, the organization requires a health-check process to assess its processes and needs to identify suitable preventive and corrective measures to achieve sustainability in the long run. We can consider audits as one such health-check tool that determines whether there is process consistency and continuous improvement and effectiveness. Some organizations conduct regular Agile assessments that look at process adherence and effectiveness.

Many organizations that comply with ISO 9001 and CMMI standards have a quality plan and are subject to a series of audits. ISO requires that organizations are audited at planned intervals so that their projects are effective and in compliance with the requirements defined in the organization's quality management system (QMS). The audit requires objective evidence to verify that the organization is complying with the set standards and guidelines.

Corrective action is a reactive approach that addresses a nonconformity that has occurred. Preventive action is a proactive approach to ensure conformance or to address the possibility of a nonconformity occurring.

Example of a Corrective Action Report (CAR)

The team is not holding retrospectives on a regular basis.

Resolution: The team was distributed, so they had difficulty holding retrospectives every sprint. They discussed the lack of retrospectives with management and decided to use an online collaboration tool for conducting retrospectives every sprint, which is quick and easy.

Example of a Preventive Action Report (PAR)

Make the new team members comfortable with adopting Agile.

Resolution: Schedule regular internal Agile refresher trainings.

Here are a few reasons for an organization to undergo an audit:
  • Compliance with the quality management system
  • Assessment of gaps that lead to corrective and preventive measures
  • Continuous improvement across all teams in the enterprise
  • Sustainability of Agile adoption
  • Process consistency across all teams in the organization
There are generally three types of audits:
  1. First-party audit: Also known as the internal audit; the company audits itself.
  2. Second-party audit: Customers audit the company.
  3. Third-party audit: An independent or external agency audits the company.
During an audit, whether it is an internal or an external audit, selected teams are audited and observations are recorded. Anything that is not in compliance with the set standards is noted as a nonconformance issue. After the audit is complete, observations are shared with the respective teams and corrective and preventive actions are planned and executed. At a later stage, the audit verifies whether corrective actions were implemented, and then the actions are formally closed. This is the general process for any organization during the audit phase.

Let us investigate how different it is to audit Agile/Scrum. An Agile/Scrum audit is called a process audit, which determines whether an organization is indeed following Agile and not just claiming to be Agile. The audit also measures the effectiveness of Scrum processes. The initial focus may be on whether the expectations are clear and to check whether the organization is compliant. If the compliance levels are good enough (management is satisfied), then the next level of auditing is recommended to determine the effectiveness of the process. In other words, does this compliance in any way help the organization further improve its Scrum processes? The audit must be tailored to suit the organization's nature of the product.

Sample audit questions:
  • How are quality objectives set for teams?
    Team's artifacts include Definition of Done.
  • How are the product planning documents archived?
    Team's artifacts include the product or sprint backlog.
  • How do you manage your daily work?
    Team's artifacts include the task board and sprint burn-down charts.
Every organization that follows Agile must also follow minimum mandatory practices. These are the process work flow steps that are mandated for all the teams to maintain consistency across the organization. These practices must be clearly defined before getting a sign-off as a mandate. The process mandates must be in synch with the organization's quality management system (QMS). The audit process determines whether these minimum mandatory processes are followed, and it will look for artifacts that support the underlying process. Exceptions are possible for a particular project in which Scrum may not be suitable. In such cases, Kanban can be used as the minimum mandatory process. The applicability of exceptions to the project must be worked out, agreed upon, and documented separately.
When an organization starts an audit for the first time, it can choose to do it in phases that first check the basic process and then check the effectiveness of the process across the entire organization. You can define the audit scope to be basic compliance in the first phase and process effectiveness in the second phase. This is just an example; organizations can begin their phases differently based on their context.

During the first phase of an audit, the organization decides to audit all the teams to determine whether they are complying with the Scrum process or any other Agile framework. In this phase, the audit focuses on practices and checks whether the teams are adhering to the set of minimum mandatory processes laid out in the QMS. The ScrumMaster plays a vital role during this phase. You can audit all the teams or only a sample of teams. The audit checks for compliance regarding Scrum events, such as sprint planning; Daily Scrum; sprint review; sprint retrospective; the Scrum team's ScrumMaster, product owner, and the development teams. The audit questionnaire clearly maps the ISO audit check points in the Scrum process.

Scrum: Describe how work is planned for a sprint.
ISO: How do you get work items and track them to completion?

Scrum: How are retrospectives conducted?
ISO: How are continuous improvements planned in your team?

Scrum: How often is the sprint burn-down chart updated, and who is responsible for the update?
ISO: How do you track work items to completion?

During the second phase, the audit focuses on the effectiveness of the process. For example, are the Scrum events timeboxed and meeting their purpose? The purpose of the audit is to determine whether the organization is really gaining a benefit from the process change. Is the process helping the organization improve its effectiveness?

The product owner checks whether the investments in the process change are worth it for the organization.

Below are sample data points that can be used in this phase to check the process effectiveness:
  • How good is the predictability?
  • How is the technical debt getting addressed?
  • Are there escaped defects?
  • What is the Say-Do ratio?
  • Is backlog grooming effective?
  • How is feedback captured and addressed during sprint reviews?
  • How are backlogs prepared?
  • Is meticulous backlog grooming happening?
  • How often are the Definition of Done and Definition of Ready revisited?
  • Is code review done in each sprint?
  • Review the automation test results, and analyze the patterns.
  • How are the retrospective action items being tracked and implemented?
During the second phase, the product owners are audited closely for their interaction with the teams to gather data points around the process effectiveness. It is a best practice to review the previous audit findings at the start of the second-phase audit. To summarize, the first phase is to check whether compliance is in place, and the second phase is to check whether the compliance is in fact giving some value to the organization.

As an internal auditor, I have seen dramatic improvements in the flow of an organization after the induction of these audit processes. Many inherent issues were brought to light and addressed across all teams. Remember, audits are a means of achieving the process improvements and not a means to check only for the compliance part! Audits are just one of the mechanisms that help with continual improvement; there could be other ways, such as Agile assessments and surveys, that support the Kaizen mindset.

Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.

Article Rating

Current rating: 4.5 (4 ratings)


Be the first to add a comment...

You must Login or Signup to comment.

The community welcomes feedback that is constructive and supportive, in the spirit of better understanding and implementation of Scrum.


Newsletter Sign-Up