Repeated CTD -- Unleashing the Potential in Multiple Testing Iterations

Test design is the process of planning and designing the tests to be performed on a software system. The time scale at which organizations perform and modify their test design is typically orders of magnitude larger than that of development, especially in modern agile development processes. As a result, often the carefully designed and optimized test suites end up being repeatedly executed with no variability between iterations, thus wasting crucial time and resources. In this work we propose a repeated test planning process based on Combinatorial Test Design (CTD), where test iterations are planned and executed while taking into account the previously executed iterations. While each iteration satisfies the same test requirements as before, we leverage the degrees of freedom in test planning in order to reach full coverage of higher levels of requirements over the course of iterations, and suggest algorithms for doing so efficiently. We evaluate our approach over a set of models collected from different sources in literature, and demonstrate significant improvement in coverage rate of higher levels, compared to the na"ive approaches.