Statistical Techniques for Performance Evaluation

Benchmarking is an integral part of the decision making process when acquiring new hardware and software. The experimental design of a benchmark trial and the methodology used to analyze the data greatly impact the resulting judgements. Mistakes made anywhere in the evaluation process are costly. Every effort should be made to ensure the validity of the judgements derived from benchmarking data. This paper discusses the benchmarking process from a statistician's point of view.