A team needs guidelines when starting to use automated tests. Not only is the development of tests expensive in terms of development time but on going maintenance of test also consume development time. One option for building a testing strategy is to use defect reports to define software use patterns. The goal is to build tests in an area that will bring the biggest bang for the buck, just like all software development.
Test Smart: When teams build tests based on reported defects many times the strategy is to build a unit test for every defect. The concept is when the team picks up a defect, they write a test for the expected behavior and see that fail. Next they write a fix that causes the test to pass. This approach will definitely increase test coverage and will yeild an automated test suite that can be used. However how do you know that you have made the best investment in writing automated tests?
Focus on Use: Similar defect reports from customers often indicate common usage of a feature that would benefit from automated tests. For example a rounding error in a monthly report. Even if there is considerable refactoring involved to build automated test for this defect, due to usage there is payback for the effort. Where a single instance of a defect or defects found by the internal quality team may indicate a feature not in heavy use so that the same refactoring effort is not going to have as big an impact on the overall code base.
Perceived Quality: Considering that only about 40 -60% of the code base is ever going to be used and that only 20 to 40 % is going to be used frequently, placing your testing efforts in the most used portion of the code is going to result in the biggest return and significantly increase customer’s perception of the product quality. Analyzing customer reported defects to generate guidelines for adding test is one tool for building higher quality code.