  Benchmarking studies in computational chemistry use reference datasets to
assess the accuracy of a method through error statistics. The commonly used
error statistics, such as the mean signed and mean unsigned errors, do not
inform end-users on the expected amplitude of prediction errors attached to
these methods. We show that, the distributions of model errors being neither
normal nor zero-centered, these error statistics cannot be used to infer
prediction error probabilities. To overcome this limitation, we advocate for
the use of more informative statistics, based on the empirical cumulative
distribution function of unsigned errors, namely (1) the probability for a new
calculation to have an absolute error below a chosen threshold, and (2) the
maximal amplitude of errors one can expect with a chosen high confidence level.
Those statistics are also shown to be well suited for benchmarking and ranking
studies. Moreover, the standard error on all benchmarking statistics depends on
the size of the reference dataset. Systematic publication of these standard
errors would be very helpful to assess the statistical reliability of
benchmarking conclusions.
