Increasingly mutable business scenarios in competitive global markets are persistently pushing decision makers and vendors toward innovative Business Intelligence (BI) solutions that should enable them to outperform competitors. BI environments are becoming increasingly complex as information needs and priorities are rapidly changing. Professionals developing these BI systems are constantly frazzled due to poor data quality, continuously changing requirements, and emerging technologies.
Many leading organizations are implementing Scrum for BI implementation for more flexible and comprehensive data discovery, context-rich analysis, and data visualization, ensuring business agility. In traditional Waterfall methods, significant time is spent detailing requirements documentation. Because of this elongated effort, in most cases, the specification documentation becomes stale well before the actual implementation begins.
To overcome this challenge, companies must replace the traditional development approaches such as Waterfall. Agile methods foster closer collaboration among business, stakeholders, and developers by promoting iterative cycles to deliver value incrementally. This collaboration consensus is realized by implementing user stories
, one of the most popular alternatives to traditional user requirement specifications. This approach allows business stakeholders to elaborate on the requirements details in a just-in-time manner, prior to the feature development.
In Scrum, business needs are captured in a single sentence by using language and terminology that business stakeholders understand. These sentences, or "stories," are then placed in prioritized order in the product backlog. The Scrum team picks up stories from the top of the list and converts these into shippable features as soon as they can.
In a typical BI implementation, the most challenging task for the Scrum team is to identify discrete workable tasks from a user story requesting a BI report. Broadly speaking, user stories are similar to the ones in traditional software development projects. However, careful analysis might uncover the following stories, or a subset, depending on various factors of a project.
Data discovery stories
The imperative steps in this story are about locating the right data. These stories might be further classified as identifying whether this data is already available and how it can be made available for the BI report. If the data is not available, more complexity is added to these stories as a way to spike the stories to investigate the data sourcing and profiling for suitability. Another possible outcome of this is data migration.
Example user story: As a supply chain analyst, I want to analyze the sales data from newly opened stores on a fortnightly basis so that I can identify inventory demand patterns for the next quarter.
Data transformation stories
In most cases, the data is available in the source system but might not be in a readily useable format. Therefore, it is essential to profile the data to evaluate the required business rules and transformations. This is accomplished in close coordination with the business user to clarify expectations. The scope of these transformations may span from simple cases (i.e., age of customer can be extracted from his/her birth date; sales profit can be calculated on costs and incomes, etc.) to more complex analysis (customer churn models).
Example user story: As a sales manager, I want customer churn rate scores so that I can make appropriate retention offers to the top 10 percent churn score customers.
Data validation stories
BI projects typically have data quality and integrity issues. Data validation stories can be used to determine whether the extracted and transformed data is of reasonably good quality. These might include straightforward stories, such as identifying missing leading or trailing zeros before decimal points or inherited data linkage issues stemming from the source system (for example, line items cannot be linked to the sale receipt). To understand data demographics and ensure that the facts and dimensions involved don't have any data integrity issues, have Scrum teams work in collaboration with business users so that the data makes more sense to the business. Identify any data anomalies and communicate them to data stewards. This way, if it is not mandated that the Scrum team address them, these anomalies can be fixed in the source system or by the upstream teams.
Example user story: As a lending officer in charge of the consumer finance department, I want email notification when loan applications without proper customer data are submitted so that I can closely monitor them to mitigate risk factors.
Data presentation and visualization stories
Finally, there are the pivotal data presentation and visualization stories, which are aimed at presenting information in a format that is easily understandable. It's possible that business users will have personal preferences about specific elements (tables, graphs, heat maps, etc.). Some of the stories might be simple, such as the creation of a matrix table or reports with summary data. Others might be more complex and prolonged due to complex analyses based on mathematical models.
Example user story: As a travel consultant, I want to know whether there is a correlation between customer age and offers bought so that I can plan more customized and effective marketing campaigns.