  In empirical software engineering, benchmarks can be used for comparing
different methods, techniques and tools. However, the recent ACM SIGSOFT
Empirical Standards for Software Engineering Research do not include an
explicit checklist for benchmarking. In this paper, we discuss benchmarks for
software performance and scalability evaluation as example research areas in
software engineering, relate benchmarks to some other empirical research
methods, and discuss the requirements on benchmarks that may constitute the
basis for a checklist of a benchmarking standard for empirical software
engineering research.
