Escaped Defects

WRITTEN BY Tom Churchwell

Escaped Defects

I was tempted to inject a typo in this post just for the irony.  If you find it, post in the comments.

Software development is challenging.

“In an agile world, baking quality in—defect prevention—is the norm, or at least it should be.”

Done well, this kind of development can be incredibly rewarding and the feedback loops are almost instantaneous.

When teams are test driving, the world of software development simply shows up differently.

“Working, Tested Product is the primary measure of progress and the value of new functionality for changes can be recognized by customers quickly”

Automation and quality is central to making that possible, but not everyone has built out Continuous Integration (CI) and Continuous Delivery (CD) such that something built today can also be tested and released to Production on the same day.

Teams that have established Definitions of Done (DoD), including code quality standards and testing standards and automation throughout, enjoy a different existence than the vast majority of teams that I have encountered.

Agile Nirvana is Rare

At least in my experience, it is. For a lot of organizations, development is separate from testing and developers may not sit with or even collaborate with testers. Work moves like a virtual assembly line from one person to the next; one environment to the next. For many organizations, the journey from development to Production (left to right) looks similar to the diagram below.

With multiple stages of testing through multiple environmental–an assembly line approach to testing–an issue can arise in any one of them. In fact, when software is not test driven, finding issues in these environments can be considered the norm.

Help! The Defects are Escaping

When things go wrong though, issues can cause organizations real discomfort. Escaped defects are a pain and it is critical to ferret them out before they get to Production. For some organizations an “Escaped Defect” is an issue that is found in Production, but I believe that sets the bar too low.  In my book, an escaped defect is any issue that is discovered after the development team is done with it. That means a separate defect is logged for anything found after it leaves the “Development Environments” depicted on the left in Green in the diagram above.

 

Escaped defects are a critical quality metric. The cost of addressing a defect increases significantly the later it is found. Remediation and then retesting are all incremental costs that basically double the cost of solution delivery. For every escaped defect, the organization is basically paying for the defect that was originally created and then paying again when it is remediated. Basically, at least twice the cost for any value that involves a defect

In my view, those costs increase in a non-linear way.  The later a defect is found, the more backflow it produces for remediation. The cost of backflow means that not just the functional code has to be rebuilt, but the test code associated with it as well. Each phase of testing may have separate test code that is affected by the changed functionality. The cost can increase exponentially depending on the extent of the defect.

The solution of course is to never produce a defect. If only the real world were that simple. Learning to test drive and evolving a DoD can help a team continually raise the quality bar and move closer to defect prevention rather than a sole reliance on defect detection. As the team learns, the DoD can change to reflect the new capabilities. Any team that is just learning TDD should not start with a quality standard of 80% automated test code coverage. It is just not realistic, and automated test code coverage may not be the most critical metric to track. Whatever metric you choose, start where you are and then increase the standard as the team develops the capability to meet it.

Here’s What You Need to Know

There are three key take-aways I want you to get from this:

  1. Traditional defect detection is an expensive way to ensure quality. Defect prevention is the ultimate goal. Just think if we could only have had today’s forward looking radar on the bow of the Titanic.
  2. Start as early as you can with Software Craftsmanship and adoption of Test Driven Development. Agile organizations are continually working toward bulletproof quality, full stack test automation, CI & CD, and the sooner you can get started the better.
  3. The goal is to not just “do Agile”, but to “be agile” and escaped defects can be used as a barometer for progress towards that agility. As the number of them goes down, agility goes up.

 

Now, Declare War on Escaped Defects

The road to agility can sometimes be long and winding. The quality of your code and your products is central to making it successfully. Declare war on your escaped defects! Track the amount of backflow you are experiencing by designating every issue that is uncovered after a team has declared a story done as an escaped defect. Live in the pain of the imperfection and use defects as a forcing function to raise the bar on quality.  Hemming and hawing about when a defect should be logged only muddies the waters. If a team said it was done and then more work gets uncovered later, declare it a defect and learn what needs to change in your process, skillset, tooling or whatever in order to prevent similar defects in the future.

Did you find the typo? I may not have had the heart to add it.

Happy Journeying.

leave a comment

Leave a comment

Your email address will not be published. Required fields are marked *

2 comments on “Escaped Defects”

  1. Greg Monahan

    The iceberg that sank the Titanic was likely not viaible on radar. Sonar would’ve been a better tool.

    Reply
  2. Chris Kaeberlein

    Found something in this paragraph, “Teams that have established Definitions of Done, (DoD) including code quality standards and testing standards and automation throughout enjoy a different existence than the vast majority of teams that I have encountered.”.

    Not sure if it’s a typo, but the “,” between “Definitions of Done” and “(DoD)” doesn’t seem correct punctuation (Should be just a space?). Then again, I was never great in English.

    Reply