Scientific Software Development and Agile
23 August 2016
Overview of the problem
The majority of scientific software development teams operate in disorganized and confusing environments. Some of the reasons for this are the complexity of the domain, lack of clarity, poorly organized resources, overcommitment to customers, colocated teams, and working in silos. Although this is typical for any scientific software development project, it leads to poor product quality, higher employee turnover, overtime, unpredictability, and team frustration. To further impair product quality, if a product owner (PO) is a non-domain or nontechnical person, the consequences are dramatic. Teams must try to resolve this problem as early as possible in the development cycle.
The three P's and a new role
We propose customized Agile practices with the three P's (perspicuity, planning, and performing), which greatly improved our performance. We also introduced a new role called the "proxy product owner" (Proxy PO), who can help bridge the gap between the team and the PO.
The PO plays an important role in Agile software development, and it's no different in scientific software development. However, in scientific software development, the PO will be either a technical person with a lack of domain knowledge, or vice versa. POs frame the requirements in their own terminology, based on their background and expertise. This creates a disconnect between the team and the PO. The problem is further exacerbated in colocated team environments, which are common in a global workforce.
To solve this problem, we introduced a new role called the Proxy PO, who will have the opposite expertise of the PO’s. We decided to use the term work flows instead of user stories, which makes a lot of sense in development and adds a new dimension when writing the requirements. Both the PO and the Proxy PO should meet frequently to write epics and work flows.
To further improve the process, the Proxy PO should be able to look into the backlog and pre-groom the stories for upcoming sprints. This will encourage more perspicuity for the team in planning, ease the development process, and help increase team velocity.
In scientific software development, a typical Scrum team will have diverse skills, including developers with or without domain skills, manual testers, and automation engineers. Sprint planning will become more tricky and lead to varying estimations, prolonged meetings without any conclusion, and unreasonable expectations.
The Proxy PO will become critical when clarifying any doubts as the team estimates work flow points. The Proxy PO will also help the team break down an epic into multiple work flows. Over time, we observed the Proxy PO taking a more active role in explaining the work flow and its impact on dependent work flows.
Even after planning, we detected some confusion between writing test cases and developing features. The confusion is more severe for long-term projects. We took a new approach by making sure design flow involved the Proxy PO and made this practice part of the Definition of Done.
The design flow can be a simple whiteboard discussion that is time bound. This helped us bridge the gap between development and testing activities and helped with bringing the work flow to a close. We also have observed an increased improvement by allowing the team to do pair programming and encouraging this practice when implementing complex work flows.
With all these practices, we were able to finish 80% of our work flows, thus meeting our Definition of Done, bolstering the team's confidence, and improving their domain understanding.
In the scientific software development industry, there is a lot of resistance to adopting Agile practices, which plays out in supporting unreasonable customer expectations and using domain complexity as a reason for teams compromising product quality. If companies such as Netflix and Google, with larger customer bases and with more complex technology, have been able to derive benefits from these practices, why not scientific companies?
Opinions represent those of the author and not of Scrum Alliance. The sharing of member-contributed content on this site does not imply endorsement of specific Scrum methods or practices beyond those taught by Scrum Alliance Certified Trainers and Coaches.
Current rating: 3.5 (2 ratings)
The community welcomes feedback that is constructive and supportive, in the spirit of better understanding and implementation of Scrum.