Lack of Predictability: Your Biggest Problem

Last Updated on Monday, 8 July 2013 10:22 Written by Andrew Fuqua Tuesday, 16 July 2013 07:13

What I’m seeing at many of my clients is an inability to know what will be completed in the next release. This shows up as missed dates and frustration over broken commitments and “unreliable teams”. When the executives inquire, they get finger pointing and non-answers, such as “the 3rd party software was late” or “software development is just inherently unpredictable”. Further, the yardstick they use to evaluate teams doesn’t help. They don’t realize, from top to bottom, that their biggest problem is a lack of predictability. Let me give you an anecdote.

I recently had the opportunity to coach a team considered the best in the company. During an earnings announcement, their CEO credited this group for driving a large percentage of the increase in revenue over the last quarter. This was a big deal. What did they do that other teams should be doing? Can they continue to deliver such results for the company? Actually, this team has no idea how much they can accomplish. Not in this coming release. Not this month. Not even this sprint. Are you seeing this in your organization?

Think Differently

They consider this the best team as measured by short-term business results, but if they aren’t predictable then revenue results aren’t enough. Although many CEOs are after short-term results, it does them no good if next quarter’s numbers aren’t close to guidance. The market hates surprises. Predictable steady growth is crucial. What this CEO really wants to know is how they are going to do next quarter.

Further, market factors and chance account for the results more than this team’s skill – any team could have produced those results given the chance to work on that product. This team actually may be the worst in the company when it comes to developing high-quality features each release. The product is the best by revenue, but the team has no influence over those results or with how they operate. Their results are a red herring.

Hidden Costs

What is the entire organization not paying attention to?  Predictability.

Here is a little taste of what’s going on: The members of this team change often. They borrow people from other teams, which contributes to their over-commitment each sprint and hurts the others. People are on multiple teams and commit to none of them. A couple days after planning, a programmer says he won’t help with their stories because he is helping another team instead. The Product Owner applies pressures to proceed without the required usability involvement, causing rework. They agree to start work on stories that are not ready for development. Attention from the directors through executives greatly increases the pressure, encouraging them to take on too much work each sprint. They regularly reschedule and ultimately cancel their retrospectives. And I could go on.

Their environment is keeping them from being predictable and productive. Sound familiar?

The Fix

Of course, it’s easy to say “don’t do that”, but much harder to implement changes. It requires a program of intentional improvement at the team, program and portfolio levels, supplemented with needs-based training, demonstrating results via agile health metrics and progress via assessment scores.

Since most organizations don’t know how to get predictable, many seek help from an enterprise-level agile coach. Since getting predictable in an enterprise is not a quick process, you need an agile coach who understands how to help when the organization isn’t agile yet. The type of coach you need will help the teams be as predictable as possible given their constraints; One who can help produce factual information about the velocity and quality that teams are able to achieve; One who can credibly express the impact of organizational behaviors that are reducing the company’s potential. You need a coach who will be a guide, interpreting assessments and metrics, relating training to day-to-day events, and cementing lessons learned in training sessions and workshops.

Our Approach

That is what my colleagues and I have been doing lately. We’ve helped companies get predictable. We focus on the end goal, but are pragmatic in our approach. Training is not enough. Sustainable change requires a holistic approach. Specifically, one must address the structure of the organization, their practices, and ultimately their culture. It takes a sustained effort to make a real difference. Engage with a company, meet them where they are, and help get them where they need to be.

* Thanks to many of my colleagues for help writing this article. I liberally stole whole sentences from Mark Kilby, Mike Cottmeyer, Dennis Stevens, Isaac Hogue and others.

Learn More

Agile Metrics : Agile Health Metrics for Predictability

Last Updated on Monday, 3 February 2014 04:37 Written by Andrew Fuqua Monday, 8 July 2013 08:00

Agile Metrics


LeadingAgile uses Agile Metrics to demonstrate the results of our process improvement efforts and to identify areas that need further improvement. We have many internal documents describing our approach that we share with our clients, but to my surprise, it seems that we have never blogged about it. Here is a high-level view of the agile metrics we often start with.

When deciding what to measure, the place to start is with a goal. First, ask yourself what outcomes are you after, your goals. Then consider what is needed to meet those goals. And finally, what agile metrics indicate whether you have what you need. You may recognize this as the Goal-Question-Metric approach.

Our clients tend to care about predictability, early ROI, improved quality, or lower cost. Predictability seems to be paramount. They want teams to get good at making and keeping promises, consistently delivering working, tested, remediated code at the end of each sprint. A team that is not predictable isn’t “bad” – but they aren’t predictable.  Without stable predictable teams we can’t have stable predictable programs, particularly when there are multiple dependencies between teams.

This post focuses on agile metrics for predictability. The goal, then, is:

Teams can plan, coordinate, and deliver predictably enough to make a release level commitment.

Here’s how we break that down:

  • Does the team deliver the functionality it intended each sprint?
  • Has the team established a stable velocity?
  • Does the team frequently deliver working, tested, remediated code?
  • Does the team have everything expected each sprint to perform the work?
  • Does the team have confidence they will deliver the functionality expected for the release?

We answer these questions with the following agile metrics:

Agile Metrics : Story and Point Completion Ratio

  • Number of Committed Stories Delivered / Number of Committed Stories
  • Number of Committed Points Delivered / Number of Committed Points

This metric helps teams become predicable in their estimating and sprint planning. It encourages smaller stories and more effort getting work ready prior to the sprint. We like to see delivered points and stories to be within 10% of the commitment.

Agile Metrics : Velocity and Throughput Variation

  • Recent Velocity / Average Velocity
  • Recent Throughput / Average Throughput

This metric helps teams become stable in their performance. This will encourage managing risks and dependencies ahead of the sprints, and not over committing within the sprint. We like to see recent velocity be within 20% of average. We also want to see a reduction in the standard deviation of the velocity over time.

Agile Metrics : Lead Time

  • WIP to Throughput Ratio

Building a large inventory of untested code typically increases the costs and time associated with fixing defects. This in turn increases the costs and challenges associated with version control, dependency management, and the delivery of working, tested, remediated code. Our objective is to improve lead-time and to deliver frequently. There should not be more than 4 weeks worth of throughput active in a team from Ready to Delivered. Less is better. We like to see 2 weeks or less.

Agile Metrics : Team-member Availability Ratio

  • Headcount available / Headcount expected

We need an indication when planned team-members aren’t available. Stability is critical for teams to be able to make and keep release commitments. When people are pulled across multiple teams – or are not available as planned – it is unlikely that the team will be able to deliver predictably. We like to see this be within 10% of plan.

Agile Metrics : Release Confidence

Use the team’s insight and record of performance to evaluate the team’s confidence that the release objectives can be achieved. This metric is useful for planning and commitment purposes. Release Confidence is a consensus vote where 1 is no confidence and 5 is very confident. If a team has heavy dependencies, they should include a vote from the Agile Project Manager of the team handling the dependencies. If the team is missing a skill or if a role is unfilled, the team should take into account the likely impact to release success. Support this metric with a release burn-up.

Other Agile Metrics

That’s just a taste of the agile metrics we use for predictability. We also use quality indicators like build frequency, broken builds, code coverage, defect rates or technical debt. Likewise, for Product Owners we are interested in things like major initiatives, features remaining, features released, size of release cycle, and more. And for value, we are interested in things like time to value.

Using agile metrics responsibly provides insight across the organization to understand the organization’s ability to meet expectations. These agile metrics help establish a shared understanding of the respective capabilities of the teams, and guidance for improvement efforts.

Learn More

How Do You Know Your Metrics Are Any Good

Last Updated on Monday, 23 September 2013 09:02 Written by Derek Huether Thursday, 4 July 2013 09:35

How to create good metricsYou want to create some metrics.  More importantly, someone has told you that you need to create some.  How do you know if you’re just making work for yourself or if you’re just putting a spin on the same old data?

Ask yourself what the goals are.

In trying to determine what to measure in order to achieve those goals, I recommend using a Goal-Question-Metric (GQM) paradigm. It can actually be applied to all life-cycle products, processes, and resources. I’ve been using this process for years and it really helps me create a quality metric, independent of processess lifecycle.

The GQM paradigm is based on the theory that all measurement should be [1] goal-oriented i.e., there has to be some rationale and need for collecting measurements, rather than collecting for the sake of collecting. Each measurement collected is stated in terms of the major goals. [2] Questions are then derived from the goals and help to refine, articulate, and determine if the goals can be achieved. [3] The metrics or measurements that are collected are then used to answer the questions in a quantifiable manner.

Here is an example of the GQM in action:

Goal 1

Maintain a maximum level of customer satisfaction

Question 1

What is the current help desk ticket trend?

Metric 1
Metric 2
Metric 3
Metric 4
Metric 5
Number of help desk tickets closed
Number of new help desk tickets open
Total number of help desk tickets open
% tickets outside of the upper limit
Subjective rating of customer satisfaction

Question 2

Is the help desk satisfaction improving or diminishing?

Metric 6
Metric 7
Metric 8
Metric 9
Number of help desk calls abandoned
Number of help desk calls answered
Number of help desk calls sent to voicemail
Subjective rating of customer satisfaction

As the great Lord Kelvin once said, “If you can not measure it, you can not improve it.”

Image based on Basili, Caldiera, and Rombach “The Goal Question Metric Approach“, 1990
Learn More

Getting Teams to Deliver Predictably

Last Updated on Tuesday, 3 December 2013 10:30 Written by Derek Huether Tuesday, 21 May 2013 03:02


queue size compared to utilization

As recently as this week, I’ve been involved in conversations with customers about how we can help make their teams deliver more predictably.  How can they meet commitments on all levels of the organization, including project, program, and portfolio?

Well, it’s not easy.  There is no silver bullet that is going to allow you to align the organization overnight.  I do, however, have one recommendation:  Stop trying to maximize the utilization of your people.  I know some are going to find that hard to understand.  To maximize value throughput, you need to keep your people as busy as possible, right?  Didn’t Henry Ford do it that way, when he had cars coming off the assembly line at three-minute intervals?  Actually, no, he did not.  What he had and what you need is a balanced system.

Getting Teams to Deliver Predictably

Henry Ford did not have everyone working at 100% utilization.  If everyone worked at 100%, the result would have been congestion — bottlenecks within his (assembly) system and the production of excessive parts inventory.  Instead, one of the many things he did was focus on limiting lead times.  That’s the time something waits before an activity happens.  By understanding his system, he was able to have the right amount of people, working at the right pace, in the right sequence, in order to maximize flow (delivery through the system).

When trying to get your teams to delivery predictably at your organization, let’s look at this from a 100,000 foot view:

  • Understand Current and Potential Capability and Capacity
  • Understand the Delivery System and Establish Goals
  • Balance Capacity and Capability with delivery throughput
  • Monitor Performance

That is how you establish predictable outcomes.

Now let’s look at this with some detail.

Understand Current and Potential Capability and Capacity

You’ve probably heard the analogy of a freeway being a value delivery system.  If not, let me draw the parallels.  On a freeway, we don’t care about utilization; we care about throughput. That is, we don’t care how many vehicles can fit onto the freeway. We care how quickly we get from point A to point B.  Measuring the capacity of the freeway is not going to directly help us. Measuring the throughput will.  For those who follow Lean Startup, these are referred to as vanity metrics and actionable metrics.

Actionable metrics can lead to informed decisions and subsequent action.  Example, I know how fast the vehicles travel on a given freeway, therefore I can plan accordingly to arrive on time.  Vanity metrics show that you’re measuring things, but they really aren’t helping you. You need to measure the right things.  By measuring the capacity of a freeway and then trying to fully utilize it would be foolish.  Strangely enough, I see organizations do that with their people all the time.  They try to keep them as busy as possible.

Understand the Delivery System and Establish Goals

We don’t build bigger freeways so they can hold more vehicles.  We build bigger freeways because we’re not smart enough to figure out how to limit the size or amount of vehicles on them at the same time.  The fewer or smaller the vehicles on the highway at the same time, the faster everyone moves along.  To increase throughput (speed) on a freeway, you need to increase the ratio of space utilized by a vehicle relative to the total space of the freeway.  If we could increase the (distance) buffers between the vehicles, we’d have fewer start and stops along our commutes. Once we hit higher utilization rates, things dramatically slow down until we have traffic jams.

Balance Capacity and Capability with delivery throughput

It’s the same thing with knowledge based systems!  Exceed a 70% utilization rate and you’ll begin to see dramatic performance decreases.

One thing that I have seen that is bringing it together is enabling teams to make their own commitments.  Once they have a sequenced queue of work and all the people necessary to complete that work, allow them to commit to, start, and then finish it.  You should begin to see the flow of value start to emerge.  Don’t pull people from the team to give them “busy” work.  Don’t push extra work on the team to keep them busy.

Monitor Performance

You can tell if your people are over-utilized by measuring the lead times.  If their work is properly sequenced, and they limit the size and volume of work they agree to do at any given time, the result should be minimal delays.  If you want to go faster, you may have to change the system.  Measure how long it takes to get something through your system.  Reflect on that.  Were there any dependencies on other people or resources that slowed you down?  Did you have your people over-utilized?  Was the work you committed to too big?  Look for an area of possible improvement, address it, and run work through your system again.  Did the lead time get shorter?

Going back to the commuting analogy, for those doing the driving, understand the conditions and know the optimal start time to begin your commute in order to avoid delays and arrive at your destination without breaking any laws.  For those asking for arrival commitments, respect what the driver tells you.  If you don’t, you’ll find people doing things like driving on the shoulder or illegally speeding in the express lane, just to arrive on time.  Sooner or later, there’s going to be an accident.


Learn More

I Want Control

Last Updated on Friday, 16 November 2012 12:53 Written by Mike Cottmeyer Friday, 1 February 2008 03:05

As an Agile Project Leader, I feel like I have lost control.

In my last post “The Road to Agility”, my original premise was that in order to be trusted, we have to be trustworthy. To earn the trust of our organizations, teams have to consistently deliver. To deliver regularly, we have to have structure and control.

I wimped out. I substituted the word discipline for control because I was concerned I would lose my audience. I was afraid people would react to the word control and stop listening.

I lost control, now I want it back.

As an Agile Project Leader, why should I be scared to use the word control? As a community, are we scared to be in control of our projects? I sure hope not.

But wait… we want collaboration, not control…. right? Control suggests authority. It suggests top down management. It suggests all those outdated project strategies we have seen fail time and time again.

Maybe when we talk about control, we just need to be specific about what we are controlling. Are we controlling people or are we controlling processes? Are we trying to control using predictive methods or recognizing the need for empiricism. These things matter when we talk about being in control.

As I write this, I am on a plane to Denmark. I sure hope my pilot is in control of this aircraft. Does that mean he knows every pocket of turbulence or mechanical issue we might encounter? Of course not. I do expect him to know where he is, to measure frequently to make sure we are on track, and adjust if we are not. I have to trust he will know we are off course long before we find ourselves out of fuel and having to land short of our destination.

Control on an agile project is all about visibility, inspection, and adaptation. You need to know what you have delivered and what is left to do. You need to know where you are at all times in relation to your goal. When you adjust, you need to be able to understand and communicate the impact to where we thought we were going.

Just like the pilot relies on a skilled support team to help him stay in control of the aircraft, the agile leader needs to collaborate with the team to maintain control of the project. It is not either collaboration or control, we must have both.

Don’t be afraid to be in control of your projects. Your teams and your business owners will thank you for it.

Learn More