Skip to main content

Escaped Defects

Tom Churchwell Senior Consultant
Reading: Escaped Defects

Escaped Defects

I was tempted to inject a typo in this post on Escaped Defects, just for the irony.  If you find it, post in the comments.

Software development is challenging.

“In an agile world, baking quality in— escaped defects prevention—is the norm, or at least it should be.”

Done well, this kind of development can be incredibly rewarding and the feedback loops are almost instantaneous.

When teams are test driving, the world of software development simply shows up differently.

“Working, Tested Product is the primary measure of progress and the value of new functionality for changes can be recognized by customers quickly”

Automation and quality is central to making that possible, but not everyone has built out Continuous Integration (CI) and Continuous Delivery (CD) such that something built today can also be tested and released to Production on the same day.

Teams that have established Definitions of Done (DoD), including code quality standards and testing standards and automation throughout, enjoy a different existence than the vast majority of teams that I have encountered.

Agile Nirvana is Rare

At least in my experience, it is. For a lot of organizations, development is separate from testing and developers may not sit with or even collaborate with testers. Work moves like a virtual assembly line from one person to the next; one environment to the next. For many organizations, the journey from development to Production (left to right) looks similar to the diagram below.

Escaped Defects

With multiple stages of testing through multiple environmental–an assembly line approach to testing–an issue can arise in any one of them. In fact, when software is not test driven, finding issues in these environments can be considered the norm.

Help! The Defects are Escaping

When things go wrong though, issues can cause organizations real discomfort. Escaped defects are a pain and it is critical to ferret them out before they get to Production. For some organizations “Escaped Defects” are an issue that are found in Production, but I believe that sets the bar too low.  In my book, an escaped defect is any issue that is discovered after the development team is done with it. That means a separate defect is logged for anything found after it leaves the “Development Environments” depicted on the left in Green in the diagram above.


Escaped Defects

Escaped defects are a critical quality metric. The cost of addressing a defect increases significantly the later it is found. Remediation and then retesting are all incremental costs that basically double the cost of solution delivery. For every escaped defect, the organization is basically paying for the defect that was originally created and then paying again when it is remediated. Basically, at least twice the cost for any value that involves a defect

Escaped Defects

In my view, those costs increase in a non-linear way.  The later a defect is found, the more backflow it produces for remediation. The cost of backflow means that not just the functional code has to be rebuilt, but the test code associated with it as well. Each phase of testing may have separate test code that is affected by the changed functionality. The cost can increase exponentially depending on the extent of the escaped defect.

The solution of course is to never produce a defect. If only the real world were that simple. Learning to test drive and evolving a DoD can help a team continually raise the quality bar and move closer to defect prevention rather than a sole reliance on defect detection. As the team learns, the DoD can change to reflect the new capabilities. Any team that is just learning TDD should not start with a quality standard of 80% automated test code coverage. It is just not realistic, and automated test code coverage may not be the most critical metric to track. Whatever metric you choose, start where you are and then increase the standard as the team develops the capability to meet it.

With Escaped Defects, Here’s What You Need to Know

There are three key take-aways I want you to get from this:

  1. Traditional defect detection is an expensive way to ensure quality. Defect prevention is the ultimate goal. Just think if we could only have had today’s forward looking radar on the bow of the Titanic.
  2. Start as early as you can with Software Craftsmanship and adoption of Test Driven Development. Agile organizations are continually working toward bulletproof quality, full stack test automation, CI & CD, and the sooner you can get started the better.
  3. The goal is to not just “do Agile”, but to “be agile” and escaped defects can be used as a barometer for progress towards that agility. As the number of them goes down, agility goes up.


Now, Declare War on Escaped Defects

The road to agility can sometimes be long and winding. The quality of your code and your products is central to making it successfully. Declare war on your escaped defects! Track the amount of backflow you are experiencing by designating every issue that is uncovered after a team has declared a story done as an escaped defect. Live in the pain of the imperfection and use defects as a forcing function to raise the bar on quality.  Hemming and hawing about when a defect should be logged only muddies the waters. If a team said it was done and then more work gets uncovered later, declare it a defect and learn what needs to change in your process, skillset, tooling or whatever in order to prevent similar defects in the future.

Did you find the typo? I may not have had the heart to add it.

Happy Journeying.

Next Agile 2018 Interviews: Day Three

Comments (7)

  1. Greg Monahan

    The iceberg that sank the Titanic was likely not viaible on radar. Sonar would’ve been a better tool.

    • Tom Churchwell

      Thanks for highlighting a nuance. Valid point

  2. Chris Kaeberlein

    Found something in this paragraph, “Teams that have established Definitions of Done, (DoD) including code quality standards and testing standards and automation throughout enjoy a different existence than the vast majority of teams that I have encountered.”.

    Not sure if it’s a typo, but the “,” between “Definitions of Done” and “(DoD)” doesn’t seem correct punctuation (Should be just a space?). Then again, I was never great in English.

  3. andrew fuqua

    “multiple environmental”?

    • andrew fuqua

      “test driven” should be hyphenated.

      Capitalization of Green and Production and etc?

  4. Ignacio Inchausti

    Hi Tom – Thank you for the article. I enjoyed the simplicity of your values: “Quality. Steady, value-added work. Learning.” As an aside, I think I found the typo:

    “With multiple stages of testing through multiple environmental…”

    “Environmental” should read “environments”… perhaps.

    Kind regards – Ignacio.

  5. Michael Walker

    Welp, one typo is that you don’t have a label on your y-axis on your “cost to fix” graph. And this is because the whole graph is not based on science — it’s made up! If that’s not correct, please provide the source of underlying information — or take down the chart entirely because it’s promoting bad science.


Leave a comment

Your email address will not be published. Required fields are marked *