Some Graphical Aids for Deciding When to Stop Testing Software.

Developers of large software systems must decide how much to test a piece of software before it is released. We consider an explicit tradeoff between the costs of testing and releasing. The cost of testing may include the cost of a lost economic initiative because of continued testing and the cost of releasing may include the cost of customer dissatisfaction and cost of fixing an unknown number of faults in the released version. The problem was formulated by the authors as a sequential Bayes problem (J. Am. Statist. Assn., 1988) in which information from past experience is also used. The limiting form of this policy when there are a large number of faults was obtained. The asymptotic distribution of the remaining number of faults upon stopping was derived. However, in that paper we made a crucial assumption, namely that the distribution of the fault finding rate is known. This happens when the testing is an ongoing activity, where the modules that are being tested are similar in design, architecture, and are developed and tested by the same development and testing groups, respectively. In this paper, we consider two important variants, where the fault finding distribution is not completely known. They are: i) the distribution, though unknown, is fixed over time, and ii) the distribution is smoothly varying over time. We modify our procedure to allow for dynamic updating of the distribution for both of these cases. For using these procedures we propose several new plots which are very easy to implement. We illustrate these plots in context of data from testing a large telecommunications software system.