Agile vs. Waterfall

WRITTEN BY Dennis Stevens

Agile versus Waterfall

These words have become completely overloaded when discussing product development. Lots of conversations about helping organizations improve their product development processes go sideways based on individual perspectives about the meaning of Waterfall and Agile. At this point these words don’t provide a distinction that is helpful when we are trying to figure out how to build products in organizations. It’s a red-herring contradiction. When I hear someone say “we need to do that Waterfall” or “that wouldn’t work in Agile” or “we can use Agile for that project” then I want to stop and ask “what does Agile mean to you?” or “what do you mean by Waterfall?”

I want to break down this debate into a clearer discussion around specific characteristics; Type of Effort, Upfront Planning, Sequencing and Feedback, and Composition of Teams. Then talk about how understanding these characteristics helps us improve the ability to be predictable and achieve the fastest time to ROI.

Type of Effort

Product Development encompasses different types of efforts. Agile is frequently viewed as the model for doing exploratory product development. In exploratory projects, teams are exploring new ways of solving a problem or trying to discover a new customer or market segment. Mike Cottmeyer has been calling these divergent projects. Fast time to value involves rapidly performing lots of experiments, getting feedback on your hypothesis, and adapting your approach.

At the other end of the spectrum are convergent projects. In a convergent project – there is a scope, a due date, and a budget. Agile is a very strong approach for delivering convergent projects. Organizations doing convergent projects need teams to have a pretty strong ability to make and meet commitments and they have a significant need to manage risks and coordinate dependencies.  These organizations can make the trade-off decisions necessary to maximize the chances they will converge on the best outcome.

Most of the organizations we deal with need to have a pretty clear idea about what they are going to get — and when they will get it — before they decide to do a project. They are primarily performing convergent projects. You manage convergent and divergent projects differently. A frequent challenge we see is when organizations move to Agile, they treat all projects like divergent projects but they are actually in convergent projects.

Upfront Planning

Not planning isn’t an option. The question is how much planning do you need to do to be successful? It isn’t possible to be predictable if you don’t do a sufficient amount of planning. The red-herring in Agile is that you don’t do any planning. The Red-herring in Waterfall is that you get all planning done up front. Regardless of your approach to the project the amount of upfront planning you do should be based on how much you need to do to answer the questions of, “What will I get?” and “When will it be done?”.

Sequencing and Feedback

As I stated above, most organizations need a pretty clear understanding of what they are going to get and when they will get it. We see different modes of sequencing and feedback. Many organizations accomplish predictability by sequencing the work in phases. For example, Analyze, Architect, Design, Construct, Integrate, and Test/Remediate.  The intention is to make good decisions upstream that protect downstream efforts.

The problem that we see in these projects is that testing and integration feedback is deferred until late in the project. Challenges identified late put the project under duress at a point when it is difficult to respond effectively. This is the project that takes as long to complete the last 20% as it took to deliver the first 80%. In phase based projects it is very difficult to be predictable because risks tend to manifest late in a project – and time to ROI is negatively impacted when project durations run past the planned delivery.

The other form of phasing and sequencing leverages frequent delivery of working tested product. When we have done sufficient planning, frequent deliveries can be sequenced to maximize the potential for success on the project. Risks and dependencies are resolved early – while there is time to respond to them. More valuable work is delivered earlier in the project – providing the opportunity to explore options and stop when the problem is sufficiently solved. We gain insight from frequent deliveries and this insight is used to make trade-off decisions to maximize return on the project. Projects using value based sequencing can be more predictable than phase based projects. Time to ROI can also be maximized in value based sequencing in those cases where a smaller subset of the potential set of features can be deployed to deliver value.

Composition of teams

In most organizations, we see people belong to functional groups and are matrixed into various project teams. It makes sense when the sequencing is in sequential phases. Individuals are often spread across multiple projects simultaneously. Using this method, we can maximize the utilization of everyone in the organization.

In another model, people belong to collaborative teams that work together to deliver working, tested product frequently. In this approach, team members have almost instant availability to each other and everything needed to deliver its next increment.  People are generally dedicated to a single team so they can work collaboratively and with near immediate availability on each frequent delivery. Co-located teams help with this immediate availability.

Team composition plays a crucial role in the ability to leverage value-based sequencing. The collaborative team composition approach may have a negative impact on utilization (I disagree with that assumption, and I’ll tell you why in a follow up blog post next week). The value gained from increased predictability and faster time to ROI often outweighs the impact of lower utilization.

Summary

Arbitrary references to Waterfall and Agile don’t provide much useful information. To be as predictable as possible and to maximize time to ROI, we need to have clear conversations around the characteristics of the teams that are best suited to the problems we are trying to solve. Understand if your project is convergent or divergent, determine the appropriate amount of planning, assess if your project would benefit from phase based or value based sequencing, and whether functional separate or collaborative teams are the best way to build software.

In my experience, Agile is about collaborative teams and value based delivery. I prefer this approach on divergent projects that require little governance and upfront planning, and on convergent projects that require significant governance and upfront planning. So I am very comfortable with Agile – whether the project is a divergent or a convergent type of effort and regardless of the appropriate amount of upfront planning.  Not everyone sees these the same way because not everyone has the same perspective of Agile. Let’s move away from red-herring conversations of Agile versus Waterfall and focus on meaningful characteristics.

leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

11 comments on “Agile vs. Waterfall”

  1. Jardena

    Great article! This framework will keep countless conversations from spinning off course in our organization. We struggle daily with preconceptions about the word “Agile” to the point where we avoid the “A” word and replace it with specific practices or principles.

    Reply
  2. Adam

    I’ve always thought the main disconnect has been that “Agile” describes a way of thinking whereas Waterfall is more of a specific process. My feeling is that Waterfall processes could be part of an Agile implementation. In fact, our project manager has long run projects with constant feedback, even while in a strict waterfall environment.

    Reply
  3. lee

    Not planning isn’t an option. The question is how much planning do you need to do to be successful? It isn’t possible to be predictable if you don’t do a sufficient amount of planning. The red-herring in Agile is that you don’t do any planning.

    Maybe you meant, “in Agile is that you don’t do any upfront planning.” Otherwise you have a paradox.

    Reply
    • Dennis Stevens

      Thanks Lee. I am trying to create a distinction between how much and when you do planning and the term Agile. My point is that there is a perception that Agile means no planning – upfront or otherwise. We need to make situation appropriate decisions about what the right amount of planning is – and when it is best done.

      Reply
  4. Dennis Stevens

    Adam. At its simplest distinction – Agile thinking involves validating frequently – Waterfall thinking involves validating late. Everything else is on a continuum and is subject to these two underlying objectives.

    Reply
  5. Tim

    i like that distinction as I had always considered Agile to be about making it up as you go along and less suited toward convergent projects

    Reply
  6. Elena Yatzeck

    This is an excellent and pragmatic way to talk about your SDLC!

    If it helps, I have taken to describing Agile as a philosophy which serves as an umbrella term for a lot of different methodologies which have their own names, rules, and merchandising revenue streams. Agile is existential, whereas methodology description can be either normative or descriptive, depending on whether you’re teaching a new team, or just providing tips to an experienced one.

    Cheers, Dennis!

    Reply
  7. Dennis Stevens

    Elena,

    Good to hear from you. I agree with how you parse the meaning of Agile. I like the distinction of normative or descriptive. But this also goes to my point. Most organizations we deal won’t benefit from existential and/or philosophical discussions of what is or isn’t Agile and Agile vs Waterfall. We need to be very pragmatic to move most organizations to action.

    Let’s catch up soon.

    Reply
  8. Brad Kekst

    Excellent article, great to hear someone get into specifics and not be religious about either methodology.

    In the fourth paragraph is there any chance you transposed Agile for Waterfall (“In a convergent project – there is a scope, a due date, and a budget. Agile is a very strong approach for delivering convergent projects.”)?

    Reply
  9. Abhishek Sen

    In terms of team composition, I have seen an issue with long running Agile projects which I have not experienced in waterfall. Consider an Agile project running for multiple years. Developers gain experience and mature, and expectations change and often managers are faced with the dillema of keeping the agile teams intact and making room for developers to grow and take on technical leadership roles. In a waterfall scenario, generally there are specified roles like Technical / Project lead / Architects – who perform somewhat different roles, and high performing developers with potential can aspire towards these. However in Agile there are no such roles, and over time the senior developers question thier career path and potential for growth in an Agile environment.

    Reply