Agile versus Waterfall
These words have become completely overloaded when discussing product development. Lots of conversations about helping organizations improve their product development processes go sideways based on individual perspectives about the meaning of Waterfall and Agile. At this point these words don’t provide a distinction that is helpful when we are trying to figure out how to build products in organizations. It’s a red-herring contradiction. When I hear someone say “we need to do that Waterfall” or “that wouldn’t work in Agile” or “we can use Agile for that project” then I want to stop and ask “what does Agile mean to you?” or “what do you mean by Waterfall?”
I want to break down this debate into a clearer discussion around specific characteristics; Type of Effort, Upfront Planning, Sequencing and Feedback, and Composition of Teams. Then talk about how understanding these characteristics helps us improve the ability to be predictable and achieve the fastest time to ROI.
Type of Effort
Product Development encompasses different types of efforts. Agile is frequently viewed as the model for doing exploratory product development. In exploratory projects, teams are exploring new ways of solving a problem or trying to discover a new customer or market segment. Mike Cottmeyer has been calling these divergent projects. Fast time to value involves rapidly performing lots of experiments, getting feedback on your hypothesis, and adapting your approach.
At the other end of the spectrum are convergent projects. In a convergent project – there is a scope, a due date, and a budget. Agile is a very strong approach for delivering convergent projects. Organizations doing convergent projects need teams to have a pretty strong ability to make and meet commitments and they have a significant need to manage risks and coordinate dependencies. These organizations can make the trade-off decisions necessary to maximize the chances they will converge on the best outcome.
Most of the organizations we deal with need to have a pretty clear idea about what they are going to get — and when they will get it — before they decide to do a project. They are primarily performing convergent projects. You manage convergent and divergent projects differently. A frequent challenge we see is when organizations move to Agile, they treat all projects like divergent projects but they are actually in convergent projects.
Not planning isn’t an option. The question is how much planning do you need to do to be successful? It isn’t possible to be predictable if you don’t do a sufficient amount of planning. The red-herring in Agile is that you don’t do any planning. The Red-herring in Waterfall is that you get all planning done up front. Regardless of your approach to the project the amount of upfront planning you do should be based on how much you need to do to answer the questions of, “What will I get?” and “When will it be done?”.
Sequencing and Feedback
As I stated above, most organizations need a pretty clear understanding of what they are going to get and when they will get it. We see different modes of sequencing and feedback. Many organizations accomplish predictability by sequencing the work in phases. For example, Analyze, Architect, Design, Construct, Integrate, and Test/Remediate. The intention is to make good decisions upstream that protect downstream efforts.
The problem that we see in these projects is that testing and integration feedback is deferred until late in the project. Challenges identified late put the project under duress at a point when it is difficult to respond effectively. This is the project that takes as long to complete the last 20% as it took to deliver the first 80%. In phase based projects it is very difficult to be predictable because risks tend to manifest late in a project – and time to ROI is negatively impacted when project durations run past the planned delivery.
The other form of phasing and sequencing leverages frequent delivery of working tested product. When we have done sufficient planning, frequent deliveries can be sequenced to maximize the potential for success on the project. Risks and dependencies are resolved early – while there is time to respond to them. More valuable work is delivered earlier in the project – providing the opportunity to explore options and stop when the problem is sufficiently solved. We gain insight from frequent deliveries and this insight is used to make trade-off decisions to maximize return on the project. Projects using value based sequencing can be more predictable than phase based projects. Time to ROI can also be maximized in value based sequencing in those cases where a smaller subset of the potential set of features can be deployed to deliver value.
Composition of teams
In most organizations, we see people belong to functional groups and are matrixed into various project teams. It makes sense when the sequencing is in sequential phases. Individuals are often spread across multiple projects simultaneously. Using this method, we can maximize the utilization of everyone in the organization.
In another model, people belong to collaborative teams that work together to deliver working, tested product frequently. In this approach, team members have almost instant availability to each other and everything needed to deliver its next increment. People are generally dedicated to a single team so they can work collaboratively and with near immediate availability on each frequent delivery. Co-located teams help with this immediate availability.
Team composition plays a crucial role in the ability to leverage value-based sequencing. The collaborative team composition approach may have a negative impact on utilization (I disagree with that assumption, and I’ll tell you why in a follow up blog post next week). The value gained from increased predictability and faster time to ROI often outweighs the impact of lower utilization.
Arbitrary references to Waterfall and Agile don’t provide much useful information. To be as predictable as possible and to maximize time to ROI, we need to have clear conversations around the characteristics of the teams that are best suited to the problems we are trying to solve. Understand if your project is convergent or divergent, determine the appropriate amount of planning, assess if your project would benefit from phase based or value based sequencing, and whether functional separate or collaborative teams are the best way to build software.
In my experience, Agile is about collaborative teams and value based delivery. I prefer this approach on divergent projects that require little governance and upfront planning, and on convergent projects that require significant governance and upfront planning. So I am very comfortable with Agile – whether the project is a divergent or a convergent type of effort and regardless of the appropriate amount of upfront planning. Not everyone sees these the same way because not everyone has the same perspective of Agile. Let’s move away from red-herring conversations of Agile versus Waterfall and focus on meaningful characteristics.