  Statistical significance tests can provide evidence that the observed
difference in performance between two methods is not due to chance. In
Information Retrieval, some studies have examined the validity and suitability
of such tests for comparing search systems. We argue here that current methods
for assessing the reliability of statistical tests suffer from some
methodological weaknesses, and we propose a novel way to study significance
tests for retrieval evaluation. Using Score Distributions, we model the output
of multiple search systems, produce simulated search results from such models,
and compare them using various significance tests. A key strength of this
approach is that we assess statistical tests under perfect knowledge about the
truth or falseness of the null hypothesis. This new method for studying the
power of significance tests in Information Retrieval evaluation is formal and
innovative. Following this type of analysis, we found that both the sign test
and Wilcoxon signed test have more power than the permutation test and the
t-test. The sign test and Wilcoxon signed test also have a good behavior in
terms of type I errors. The bootstrap test shows few type I errors, but it has
less power than the other methods tested.
