Why the Scrum framework is the best solution for the customer in software development

Having followed the more traditional development methodology Waterfall early on in my career, I moved into a customer facing Project Management role where I managed the implementation of bespoke solutions for each customer. As such, the customer required high levels of input into the solution and requirements often changed or were unpredictable. Waterfall was therefore not practical. I took the Scrum Master course and certification and found another way.

I don’t pretend to be an expert on the intricacies of Agile software development, but I do have a general understanding. I’d like to explore at a high level why the Scrum framework is the best solution for the customer in software development, said by a Project Manager.

Don’t go chasing Waterfall

I want to start with Waterfall. During the 1990’s it was felt by some in the industry that current software development practices were inadequate and with significant shortcomings. They were becoming increasingly frustrated with the rigidity of the Waterfall approach. In Waterfall the entire process of software development is divided into separate phases, all phases are cascaded into each other, progress is seen as flowing steadily into the next phase like a Waterfall. The phases do not overlap and the next phase is not started until the previous phase has completed. Software is delivered at the end of the process. Waterfall is simple and easy to understand, it allows for scheduling and control. For Project Manager’s it’s easy to manage and provides a clear list of deliverables, milestones, and timescales for the customer. Waterfall sounds great? Sadly not. Changes to requirements in a Waterfall project are disruptive, change is almost impossible once the testing phase has been entered and a change in scope can end the project. Waterfall is open to high risk and uncertainty as the user only sees the software at the end of the process, which means that the project may get discontinued if requirements are not understood correctly at the beginning, and there may be a need to start all over again.

The main issue with Waterfall is that it doesn’t allow for a change in requirements until right at the end of the process after the solution had been delivered to the customer, so the solution often isn’t fit for its new purpose as requirements frequently evolve and change over time. For longer projects this doesn’t work as these projects can take years, they’re expensive, and if the end result is not useable for the customer, this results in a significant loss of time, effort and money. For Waterfall to work, requirements need to be fully understood at the start of the project, documented, and not subject to change. Unrealistic right?

The birth of Agile

In 2001, 17 thought leaders from the software development industry spent time together at the Snowbird Ski Resort in Utah. The objective? To discuss common ground and best practices, and to eat, ski, and relax. It was here that the Agile Manifesto was born.

The term Agile was coined at this meeting and it refers to a repetitive cycle and the incremental delivery of software development. In Agile, requirements and software solutions evolve over time, frequent planning and customer feedback is regularly incorporated, and there is an ongoing alignment between technology and the business or customer. Agile is adaptive, it embraces and manages changes in requirements and customer priorities.

Agile favours Individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. Agile sounds great! Sign me up.

Purist Agile isn’t welcome at a highly regulated party

However, put Agile into an environment where everything is regulated, high levels of governance and documentation are required for audit, and you’ve got yourself every Project Manager’s worst nightmare. With purist Agile, it’s difficult to assess the total effort required and cost of the project at the beginning of the software development lifecycle. It can also be incredibly demanding on the users’ time and requires a big commitment from the stakeholders when the project isn’t their day job. I’m in agreement, Waterfall definitely has its shortcomings. For longer projects, Waterfall is unworkable. It’s not advisable to spend years on a project, a big budget, when there’s the stipulation that no one can change their mind until the end of the project. However, I’d argue that elements of purist Agile share similar weaknesses. In a perfect world, Agile is perfect. However, bring to the table lightweight planning, the impracticality of the customer being constantly involved in the process, a lack of documentation given that most conversations with customer are verbal, a lack of contractual commitments on cost and time and you’ve got yourself a nightmare for the customer. Furthermore, when implementing Agile company leaders really need to buy in to Agile and to understand it which often isn’t the case. I believe an Agile mentor or coach for the leadership team and managers is essential, and people need to be on board with its limitations as much as they are on board with its benefits. For example, rapid release cycles of useable software are great, but how valuable is this without a prediction at the start of when the final product is going to be delivered. In pure Agile developers don’t give total estimates. From my experience one of the customer’s first question’s is when will it be delivered and how much will it cost? With Agile, it’s difficult to assess the total effort required at the beginning of the software development lifecycle and so we’re generally unable to provide full timelines and by association give an indication of how much it will cost at the end.

Surely there has to be another way? There is. Scrum please take a seat at the table.

Scrum – an ordered formation of players working towards the same goal

Scrum’s story actually started in 1986 when Hirotaka Takeuchi and Ikujiro Nonaka introduced the term Scrum in the context of software development in the Harvard Business Review publication, “New New Product Development Game.” In 1993, Jeff Sutherland and Ken Schwaber began the first Scrum project, and in 1995 Scrum was developed as a formal framework and was presented at the OOPSLA Conference in Texas. Jeff and Ken were one of the seventeen thought leaders present at the Snowbird ski resort in 2001.

Scrum is an iterative and incremental agile software development framework for managing product development. It defines “a flexible, holistic product development strategy where a development team works as a unit to reach a common goal.”

The great thing about Scrum, said by a Project Manager, is that it gives a more formal framework around purist Agile methodologies. In Scrum, at the start of each development cycle, known as a “sprint,” the sprint planning happens, and this meeting allows the customer to know what’s going to be delivered at the end of each sprint. Additionally, each sprint is for a set period of time (Yes! Timelines,) and this is usually 2-4 weeks. At the end of the cycle comes the Sprint Review where the development team show what they have completed during the sprint. This meeting is a demo attended by the Scrum Team and the business stakeholders or customer, and presents the features that have been developed.

For me, Scrum complements both Waterfall and purist Agile in that it allows for timelines, estimates, and change requests in a controlled manner (change requests are encouraged in Scum but they must take place either side of each sprint.) Scrum emphasises the need for self-organising teams. Scrum provides the framework and the Scrum Team is then trusted and empowered to come up with ‘how’ and the ideas. Developers are most likely to favour Scrum over Waterfall as it allows for more autonomy. What’s great about Scrum is that it’s modifiable to match your own set of circumstances. It’s adaptive. For the bulk of projects, documentation and planning is required, but it does need to be flexible to account for changes in requirements. Scrum allows for this. Scrum suits projects with aggressive deadlines, and complex requirements. This is every customer’s dream right? Their input into the Product features is encouraged, there is set timelines for delivery, and at the end comes a base version of the product, useable software or features, which can be built on over time, resulting in early feedback from the customer and a return on investment. Winner.

Scrum allows for more structure around purist Agile methodologies but at the same time, it lessens the need for onerous business process documents and project documentation often required with the Waterfall approach.

As with all Agile methodologies, Scrum recognises the need for regular customer input and feedback, it welcomes controlled changes outside of the sprints, and provides an opportunity to deliver useable software that meets business and customer requirements. I’m not saying that Waterfall doesn’t have its place, it does, but Scrum seems to be the best solution for the customer in software development.