Agile Metrics : Agile Health Metrics for Predictability

Written by Andrew Fuqua Monday, 8 July 2013 08:00

Agile Metrics

 

LeadingAgile uses Agile Metrics to demonstrate the results of our process improvement efforts and to identify areas that need further improvement. We have many internal documents describing our approach that we share with our clients, but to my surprise, it seems that we have never blogged about it. Here is a high-level view of the agile metrics we often start with.

When deciding what to measure, the place to start is with a goal. First, ask yourself what outcomes are you after, your goals. Then consider what is needed to meet those goals. And finally, what agile metrics indicate whether you have what you need. You may recognize this as the Goal-Question-Metric approach.

Our clients tend to care about predictability, early ROI, improved quality, or lower cost. Predictability seems to be paramount. They want teams to get good at making and keeping promises, consistently delivering working, tested, remediated code at the end of each sprint. A team that is not predictable isn’t “bad” – but they aren’t predictable.  Without stable predictable teams we can’t have stable predictable programs, particularly when there are multiple dependencies between teams.

This post focuses on agile metrics for predictability. The goal, then, is:

Teams can plan, coordinate, and deliver predictably enough to make a release level commitment.

Here’s how we break that down:

  • Does the team deliver the functionality it intended each sprint?
  • Has the team established a stable velocity?
  • Does the team frequently deliver working, tested, remediated code?
  • Does the team have everything expected each sprint to perform the work?
  • Does the team have confidence they will deliver the functionality expected for the release?

We answer these questions with the following agile metrics:

Agile Metrics : Story and Point Completion Ratio

  • Number of Committed Stories Delivered / Number of Committed Stories
  • Number of Committed Points Delivered / Number of Committed Points

This metric helps teams become predicable in their estimating and sprint planning. It encourages smaller stories and more effort getting work ready prior to the sprint. We like to see delivered points and stories to be within 10% of the commitment.

Agile Metrics : Velocity and Throughput Variation

  • Recent Velocity / Average Velocity
  • Recent Throughput / Average Throughput

This metric helps teams become stable in their performance. This will encourage managing risks and dependencies ahead of the sprints, and not over committing within the sprint. We like to see recent velocity be within 20% of average. We also want to see a reduction in the standard deviation of the velocity over time.

Agile Metrics : Lead Time

  • WIP to Throughput Ratio

Building a large inventory of untested code typically increases the costs and time associated with fixing defects. This in turn increases the costs and challenges associated with version control, dependency management, and the delivery of working, tested, remediated code. Our objective is to improve lead-time and to deliver frequently. There should not be more than 4 weeks worth of throughput active in a team from Ready to Delivered. Less is better. We like to see 2 weeks or less.

Agile Metrics : Team-member Availability Ratio

  • Headcount available / Headcount expected

We need an indication when planned team-members aren’t available. Stability is critical for teams to be able to make and keep release commitments. When people are pulled across multiple teams – or are not available as planned – it is unlikely that the team will be able to deliver predictably. We like to see this be within 10% of plan.

Agile Metrics : Release Confidence

Use the team’s insight and record of performance to evaluate the team’s confidence that the release objectives can be achieved. This metric is useful for planning and commitment purposes. Release Confidence is a consensus vote where 1 is no confidence and 5 is very confident. If a team has heavy dependencies, they should include a vote from the Agile Project Manager of the team handling the dependencies. If the team is missing a skill or if a role is unfilled, the team should take into account the likely impact to release success. Support this metric with a release burn-up.

Other Agile Metrics

That’s just a taste of the agile metrics we use for predictability. We also use quality indicators like build frequency, broken builds, code coverage, defect rates or technical debt. Likewise, for Product Owners we are interested in things like major initiatives, features remaining, features released, size of release cycle, and more. And for value, we are interested in things like time to value.

Using agile metrics responsibly provides insight across the organization to understand the organization’s ability to meet expectations. These agile metrics help establish a shared understanding of the respective capabilities of the teams, and guidance for improvement efforts.


6 Comments

  1. Lack of Predictability: Your Biggest Problem | LeadingAgile   |  Tuesday, 16 July 2013 at 7:13 am

    [...] program and portfolio levels, supplemented with needs-based training, demonstrating results via agile health metrics and progress via assessment [...]

  2. Bottom-Up Implementation & Top-Down Intent | LeadingAgile   |  Tuesday, 23 July 2013 at 7:15 am

    [...] as they can within their constraints. It means getting them mature enough to report out significant performance information like velocity, story ratios, resource shortages, quality and blocking issues on a reliable cadence. [...]

  3. Dave Speck   |  Thursday, 05 September 2013 at 8:51 am

    Can you provide an example of how WIP and Throughput Ratio are calculated? Thanks.

  4. Andrew Fuqua   |  Thursday, 05 September 2013 at 12:31 pm

    Hey Dave. Thanks for the question. Metrics sound simple on paper, but putting them into practice requires some thought. And you must change them over time.

    We’re clearly talking about Little’s law here, so the simple answer to your question may be:
    Lead Time = WIP / Throughput
    Lead Time = 2 stories / 5 stories per week = .4 weeks
    Lead Time = 20 story points / 20 story points per sprint = 1 sprint

    For an iterative process, Throughput is average Velocity (for the unit of iteration time), and WIP is the number of Points in the Stories not yet Delivered. You could alternatively just use a count of stories: on average, how many stories do we deliver every n weeks, and how many stories have we started but are not yet delivered?

    Your specific situation affects how you use this metric, and remember that we’re measuring to get information so that we can make wise improvements — not for rewards or as some absolute goal. I’ll give some examples…

    Suppose we have a well-oiled Scrum team of 8 people with max WIP during their sprints at 4 stories started but not done, no carry-over to the next sprint and 16 stories in the sprint backlog and an average velocity of 16. This team doesn’t need this metric to inform them about their in-sprint WIP.

    With a very mature team that is able to deploy to production frequently, you may be more interested in the amount un-deployed work (with un-deployed work as your WIP, your definition of Delivered). If my throughput of deployed stories is 5 per week, but I have 30 stories in process, that is, undelivered (and the trend is stable or worsening), then maybe I should investigate how to decrease WIP, increase throughput, shorten lead time.

    However, consider a struggling team in which test can’t keep up with development — perhaps QA can do 10 stories/week but dev can do 11. At the end of week 1 they have 1 untested story and a throughput of 10 stories/week (fully tested) — they have a little WIP hanging out compared to their throughput. But by the end of week 10 that carryover has grown to 10 and their throughput (through QA) is also still 10. The trend was bad from the start, which I could see from week 2. That’s why I always also look at the trends of these metrics, not just the absolute number. If week 10 is the end of the release and we have 10 stories not tested, we could catch up pretty quickly if we stop developing for a bit, or increase swarming, or change how we test, or have increased quality, or whatever. But if I’ve got 4 weeks worth of un-tested stuff, then I’ve got a much bigger problem.

    In that situation, my definition of Delivered may just be tested stories, not yet packaged into a release, or not yet deployed into production. Once that team solves that current problem and starts deploying to production frequently, they’ll need to change this metric.

    I see the same kind of dev-outrunning-qa situation in iterative teams as well. When doing the math for an iterative team you need to know whether the team is counting the un-tested stories in their velocity. They /shouldn’t/, but what they /should do/ is irrelevant if you are going to re-use their measurement. What they /actually count/ matters.

    Does that help?

  5. Dave Speck   |  Saturday, 07 September 2013 at 7:03 pm

    yes, thank you

  6. Top 10 Agile Blog Posts of 2013 | LeadingAgile   |  Wednesday, 01 January 2014 at 8:01 am

    […] #2 Agile Health Metrics for Predictability by Andrew Fuqua […]

Leave a Comment