Get certified - Transform your world of work today

Close

Agile in Data Warehousing Environments

Our experience in adopting Agile in data warehousing projects

13 March 2017

Alok Kumar
Tata Consultancy Services

Suganthi Subramanian
Tata Consultancy Services


While Agile and Scrum have been widely adopted as primary methods for delivering software, people still view these frameworks with skepticism with regard to business intelligence and analytics projects, especially as solutions to meeting data integration needs. Generally, people involved in such projects find it difficult to apply the concepts of Agile/Scrum, because applying evolutionary techniques to these projects requires a radical shift in thinking. This article is a brief write-up about our experiences in adopting Agile/Scrum practices in a data warehousing (DW) project.
 

Where we stood

The figure below shows the traditional process of delivering a DW project. It aligns well with a typical Waterfall life cycle, which progresses in fixed phases.



The end-to-end solution (i.e., from creating the integrated data platform to creating reports to facilitate analytics over the platform) is built in phases, with each phase completed by teams specializing in each skill. This phased approach necessitates handoffs between the teams, resulting in efforts wasted by team handshakes after each handoff.
 
Primary Provider Primary Receiver Primary Tasks Functional Teams
Business users/SMEs Report decomposition team Provide report requirements, SME on reports/sources Team 1 (Business users)
Report decomposition team Report unwinding team Map report requirements to attributes in existing reports on legacy reports and provide SME and requirements for new reports Team 2 (Report decomp. team) 
Unwinding team Business/Data analyst Reverse engineer report attributes traversing to sources through singular/multiple hoops Team 3 (Unwinding team) 
Business/Data analyst ETL developer, data modeler Create BRD for new reports on the new environment Team 4 (Business/Data analyst team)  
Business/Data analyst ETL developer, data modeler Create FSD Team 4 (Business/Data analyst team)  
Business/Data analyst ETL developer, data modeler Create source-to-target mapping documentation Team 4 (Business/Data analyst team)  
Modeler ETL developer Create CDM, LDM, PDM Team 5 (Modeling team) 
ETL developer BI Create code to move data from sources to required target (Source to Staging to Integration to Semantics)  Team 6 (ETL development team)
BI QA Use data structures on semantics to build report framework  Team 7 (QA team)
QA Business users/SMEs (UAT) Validate reports and sign-off  Team 8 (UAT team)
Business users/SMEs (UAT) Deployment operations team Verify reports and sign-off for PROD movement Team 1 (Business users) 
Deployment operations team   Release code to PROD Team 9 (Release management team) 
 

Challenges with our Waterfall process

As illustrated in the table above, there were multiple teams (nine in this case) collaborating to deliver the product. The teams were not cross-functional and were each created to do only one specialized functional job. Each team handed off its deliverable to the next team in the work flow process. This is typical of Waterfall. However, the structure generated the following challenges:
  • The teams did not have a clear big-picture understanding of the release deliverable and were focused on their own siloed deliverables.
  • Business involvement was limited in the initial requirements and final UAT stage. They did not have much visibility into what was being developed in the interim stages. The lack of involvement limited the interactions with business, resulting in fewer opportunities to incorporate feedback into the product.
  • Handoffs required a lot of unnecessary documentation, which was of little value from a maintainability perspective.
  • Teams strongly resisted any changes initiated by the business.
  • Defects were often found too late in the process, and fixing such defects became expensive.
  • The projects often did not meet customer expectations, were late, and went over budget.
  • Coordination across these multiple teams was a challenge and required project managers to create detailed work breakdown structures.


Agile Adoption: Where we stand now

As our organization explored and then pushed Agile methods, we also wanted to leverage this opportunity to apply these principles to our own projects. After some initial training, we found, to our surprise, that the business was very much on board in experimenting with Agile practices. This was a good start for us. The more difficult part was creating Scrum Teams that were cross-functional, colocated, dedicated, and long-lasting.

There were many discussions around this initiative, and with the help of leadership support and Agile coaches, we identified one pilot project to prototype. We identified the various roles (product owner, ScrumMaster, and the team) and did an initial kick-off, followed by Sprint 0. We tweaked our process from Waterfall to the one highlighted below. We leveraged the Scrum Guide and XP practices as much as we could, followed by on-the-job guidance from a coach. Our process looked like this:



Our process includes the following:
  • Several cross-functional Agile teams, with each team representing a core functional feature of the product.
  • Each team comprises members with expertise across the requirements of the life cycle, from requirements gathering to product deployment.
  • Team members were encouraged to develop T-shaped skills; that is, to learn cross-functional skills that may not be their core specialization.
  • The size of each team is from three to nine members.
  • Each of the core deliverables is created through the collaboration of all team members.
  • Each member of the team has the same responsibility toward the quality of each deliverable as the team.
  • Business was engaged with the team; the product owner helped us get feedback faster.
  • Team collaboration was driven by daily stand-ups and Scrum of Scrums.
We realized that the key to success is to think in terms of small increments that would provide business value and to develop the solution iteratively and incrementally. We faced some early resistance because most team members were new to the Agile approach.

These are some of the challenges we faced and how we addressed them:
  • It is impossible to define a minimum viable investment (MVI) without a project that includes a UI component. The MVI is defined by identifying the smallest deliverable that the business can evaluate and provide feedback on. The MVI need not be a full report but something that the business can appreciate and react to. The end solution is built incrementally.
  • The architecture of a DW that involves multiple data ingestion layers makes Agile adoption difficult. Agile adoption across multiple data ingestion layers could be equated to the multi-tier architecture that is common in UI-based projects. MVI must be customized to suit the data ingestion layers of DW.
  • We could not develop reports, ETL, or use cases without a complete data model. Teams must realize that the data model will never be final and that changes are inevitable. Our process should easily adopt to changes, and Waterfall makes it costly to accommodate changes. It's best to invest in automation, because automated functional and regression tests go a long way to build a stable system.
  • DW projects are generally large, costly, and involve several teams. Following the Agile approach to accommodate changes makes the target more unpredictable. This unpredictability is even more reason to adopt Agile. The complex nature of DW projects and its related heavy cost demands an execution model that accommodates changes to create the right platform. Also, the flexibility in accommodating changes helps in improving customer satisfaction.


Conclusion

Based on our experiences, we realized that Agile practices bring value to DW environments. Agile can help drive higher quality, reduce risk, increase collaboration with business, and ultimately deliver business value to the customer faster. The challenges we faced were mostly people-related rather than technology-related. With constant encouragement from leadership and reinforcements from the Agile SMEs, it is indeed possible to gain significant benefits by adopting these practices.

It is important to realize, however, that Agile in different environments will look different from each other, and they may not reflect the textbook representation of an Agile project. The intention is to adopt the principles of Agile and strive to improve continuously.

 

Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.



Article Rating

Current rating: 4.9 (7 ratings)

Comments

Neeraj Singh, CSM, 3/13/2017 8:52:46 PM
Golden rule: The teams should always have a clear big-picture understanding of the release deliverable.

DW projects have long turnaround time. It requires core technical members with deep expertise in his area. So, forming a cross-functional Agile team, sharing the tech expertise and getting the work done is a big achievement.
Rumesh Wijetunge, CSP,CSM,CSPO, 3/14/2017 12:20:15 AM
Good article highlighting the problems in DW and how agile can be the solution. As mentioned in the article the requirement for specialized skills and knowledge and capabilities of using specific tools is a really key point which has a direct effect on adaptation of agile practices. I guess this depends on the size of the engagement and thus the size of the team as well. Larger the team easier it would be to cross-train the team members to make them more self organizing and cross functional. There may be some limitations when the project size in terms of finances is smaller thus requiring specialized resources to get the work done.
Somnath Ghosh, CSP,CSM,CSPO, 4/20/2017 10:02:24 PM
Very relevant article. Careful attention should be given to the strategic direction of the DW Solution Architect and long term BI Objectives of the organization. This may allow a cohesive approach to be followed much like the vision.

Thank you for sharing your experience
Dennys Regalado Díaz, CSM, 9/15/2017 6:36:42 PM
Greate work!, can you give us some directions about what kind of automation tools we can use to continuous integration, testing and fast deployment. ¿we need to built our own tools or there are something that we can reuse hosted on some repository like github?

Thanks

You must Login or Signup to comment.

The community welcomes feedback that is constructive and supportive, in the spirit of better understanding and implementation of Scrum.

 

Newsletter Sign-Up

Subscribe