When feedback from your customer includes enthusiastic comments like, "This is what we've wanted for years," and your project sponsor says, "One of the greatest projects in my career!" then you know you've done something right. When this applies to a project that was at a dead end, without much trust from stakeholders and little money left only a year earlier, such results are even more surprising.
Are they even possible? How did we manage to turn a doomed project into a success with such flying colors that even the Minister got interested? The answer is Scrum.
A while ago, our Ministry (other specifics must remain anonymous) embarked on a journey to introduce a geographic information system (GIS). The idea was to geospatially enable one of its public websites. Collaborating with another Ministry, we commissioned a project to redevelop the existing system. A modern mapping site with a supporting GIS doing the heavy lifting under the hood was the goal. While it was clear this would be a journey, it turned out to be anything but a smooth sail.
But first things first. The Ministry uses the site to provide land ownership information to interested parties: the landowners themselves, the private law community, trusts, and special-interest groups. While continuously attracting a fair amount of traffic, the system was old and tedious to use, and it had fallen behind technologically. Ease of access and providing the relevant information through this website is critical to support the Ministry's goals.
The website provides land ownership information for certain types of land equaling roughly 5.5 percent of New Zealand. The Ministry's role is to act as custodian for the ownership records of this land, which has some peculiar characteristics. Often land is owned by people without their knowledge, or even by deceased people. Owners hold ownership interests under multiple different but legitimate names. Unlocking this information and providing intuitive access to it promised to be highly beneficial for everybody. Customer expectations were understandably high.
So far, so good. Unfortunately Ronald Reagan had a point when he said, "What are the nine scariest words in the English language? 'I'm from the government and I'm here to help.'" Our intention was to boost the website's value to the public users and to our business customer. However, we hit the proverbial brick wall instead.
Geospatial technology had a bit of history at the Ministry already, as previous attempts to implement a GIS had failed. GIS technology on this scale was a first for the Ministry. Of course we wanted to get it right this time, and the answer was: in-depth requirements gathering to properly understand the business need. Thorough analysis to design a robust solution. Running a tender process with careful screening to identify a capable partner for implementation. We did everything by the book, which at the time meant following the Ministry's standard Waterfall/RUP-like process.
Finally, we arrived at a point where we had comprehensive requirements, properly documented, socialized, and agreed upon. We had an abstract but cleanly designed architectural blueprint. We had also rolled up our sleeves and were keen to attack the challenge.
What we didn't have was an implementation partner. The tender process came back without responses and left us empty-handed. Curiously, nobody wanted to implement the system for us. The unofficial feedback explained it: You're asking for a Rolls Royce, but you're only offering money for a Beetle. And on top of it all, you want it implemented yesterday.
The previous project phases, the requirements gathering and write-up, design, and the tender process had taken a long time and the deadline was looming. We, however, were stuck. Our stakeholders' confidence in our ability to deliver had taken a hit. However, we all wanted this to be a success, and failure was not an option. The biggest hurdle, though, was to get approval to proceed. Why would anybody continue to flog a dead horse? Why support a project that had already proven it couldn't deliver?
Agile as savior
For every problem, there's a solution that is obvious, logical — and wrong. In our case it was the Ministry's standard Waterfall approach that was ill suited to tackle technology that we didn't have any prior experience with. It was also less than ideal to facilitate close collaboration between the business stakeholders and the development team. In a situation where everybody should have been tight as a tick, we were forced apart by the process.
Coincidentally, both the business as well as ICT were looking into Agile processes at the time. Even though no decision had been made to adopt Scrum, the GIS project looked like a good candidate on which to try the new approach. All parties recognized that the project was caught between a rock and a hard place, and small adjustments wouldn't be good enough. Additionally, an Agile process sounded particularly appealing from a risk-mitigation vantage point. The close collaboration and short cycles would keep everybody on the same page and allow frequent and continuous adjustments where necessary. It was only toward the end that we realized there was also a theme common to a lot of decisions we made, which was pivotal to our success: Less is more.
Less design, more prototyping
The project was at a critical junction. The governance body wanted proof we could deliver this time, and they wanted it quickly. What they didn't want was promises, concepts, or plans. Or, in Linus Torvalds's words: Talk is cheap. Show me the code.
There was no time to follow the traditional approach and do detailed analysis, nor would the resulting papers have been convincing enough anyway. We needed something visual, tangible, interactive, and representative of the future end result. Something with enough punch to get an "Ahhhhhh!" and buy-in from the decision makers.
The obvious solution was to build a prototype.
It turned out that developing the prototype was not a huge endeavor. The impact, however, was phenomenal! First, it showed that the technology components we had in mind could do the job. Second, it proved that we could cope with the complexity ourselves, even though GIS technology was a first for all of us. No need for an external vendor to do the job. Finally, and most important, it convinced the nontechnical decision makers not to can the project, and it gave them confidence in our ability to deliver. The prototype put a face to fancy words, and the visual interactive map gave them something they could relate to (think Google Maps). It was the breakthrough we needed.
Less process means more bang for the buck
The Ministry used to follow a document-centric and heavyweight Waterfall development process, a RUP (rational unified process) variation. Once shifted to SCRUM, the team could focus on more than simply delivering tangible output of business value. Everybody was also much more effective and motivated. Developing something because a one-size-fits-all process mandates it is one thing. Developing something because it brings the team closer to success and closer to delivering the end result to the customer is an entirely different kettle of fish! Of course, we did produce the necessary set of documentation, but in a no-frills fashion. We also developed these elements as we went, rather than in a massive up-front and productivity-stifling design exercise.
Fewer requirements mean more achievement
When something is too big to deliver, the obvious step to take is to reduce scope and abandon some requirements. That, however, proved to be difficult. How so? Well, the notion at the time was: Requirements are required, hence the name. While it's possible to prioritize, you can't just remove some requirements — all of them need to be delivered. Hmm, that attitude didn't exactly help. After extensive hemming and hawing, we decided to do some serious culling nonetheless. When the options are to deliver some but not all of the features or nothing at all, the choice isn't that hard.
So we scaled back to the genuine essentials: pretty much only the ability to overlay land blocks on a map, supported by a search function. Shaving off all the requirement fat and cutting back the sophisticated bells-and-whistles features proved to be right thing to do. It meant the team could focus on getting the essentials right. It didn't have to work with the nagging feeling of looming failure, and it didn't have to rush the implementation. The end result was a mature, no-frills solution without any technical debt. It demonstrated one thing very effectively: A mapping application is not as hairy and scary as everybody thought. If provided with a realistic target, the team of the formerly stuck project could achieve this easily.
Less complexity means quicker results and earlier stakeholder buy-in
When we trimmed the requirements back, we deliberately removed some of the most complex ones. That lifted a huge weight off our shoulders: It not only meant less development and less testing; it also removed a whole set of "How do we do this?" questions from the equation. A sacrifice well worth it, as it created another opportunity to focus on core functionality without getting distracted. Each sprint, we delivered additional tangible output and produced new features in quick succession. The reduced complexity allowed us to get away almost entirely with a component-assembly approach. We wired existing open-source building blocks with some glue code, rather than embarking on a huge bespoke development journey. Again, this meant quicker results and quicker turnaround of features after inevitably changing requirements. It also helped us deal with the stakeholder angst about our ability to deliver, because we simply demonstrated that we could do it. The net effect was not only buy-in across the board. It also created a level of enthusiasm within the business that added to the positive momentum.
Less specialization means more collaboration
GIS is definitely a specialized field, even though mapping websites have become more commonplace in recent times. The Ministry didn't have any experience in the development of GIS applications, so the obvious choice was to contract some expertise. That didn't work out, however, as New Zealand is small by comparison, and Enterprise Java developers with a geospatial twist are a rare species.
We didn't really have a choice and, against our initial plans, just formed a core team of skilled and well-rounded Java developers with a strong can-do attitude. It wasn't until much later that we realized how much stronger this setup was. A GIS specialist would have had nobody to talk to. Nobody else spoke the GIS language, hence there was nobody to rub shoulders with or use as a sounding board for ideas. Such a specialist would have run the risk of working, perhaps not in isolation, but certainly at some disconnect from the rest of the team. Likewise, the team would have struggled to understand his activities. Having had generalists performing the GIS-related tasks meant all team members collaborated intensely and continually. They were also motivated because they had the chance to pick up something new and cool.
It all tied in nicely: The prototype, together with the abandonment of the most complex GIS requirements, removed a lot of complexity and the need for specialized GIS components and skills. It was like earning compound interest: One benefit enabled another, and their positive impact added up.
Lessons learned, or why we were successful this time
Making things work and getting a project with new technology over the line doesn't require magic, but there are three key success factors that have to be considered: people, process, and technology.
Obviously, you have to have the right people on board. Without suitable skills and a can-do attitude, it's going to be a hard slog. But even a team willing to get its hands dirty cannot be effective if stifled by convoluted, bureaucratic, or simply wrong processes. Scrum, however, is the perfect harness that brings out the best in people. It's the glue that holds the other components — people and technology — together. Scrum enabled the team of generalists to collaborate efficiently and do the heavy lifting. Ultimately, it was the new process that allowed proving the technology choices were fit for purpose, something big up-front design could not have achieved. Key to the success were also the close collaboration and short feedback loops between the team and the business stakeholders, something Scrum enforced as well.
The most remarkable outcome, however, was that the business deemed the system scope-complete with a feature set that only represented about 50 per cent of what was originally targeted. The interactive approach led to a system that delivered what the users really needed. It was different from the super-sophisticated application they thought they needed. In a nutshell, we stopped development where the end product was good enough. That dovetailed nicely with the goal to implement the simplest solution that did the job. Amazingly, the business regarded this as a perfect fit for their needs! Simple, by the way, does not have to mean crude. It just says no gold-plating, no second-guessing of the business need by implementers, and no features that are there just because they are cool.
Had we followed the iterative prototyping approach straight away, we would have realized one thing: We were dealing with a nut — no need to get the sledgehammer out of the shed.