  In natural language processing, a recently popular line of work explores how
to best report the experimental results of neural networks. One exemplar
publication, titled "Show Your Work: Improved Reporting of Experimental
Results," advocates for reporting the expected validation effectiveness of the
best-tuned model, with respect to the computational budget. In the present
work, we critically examine this paper. As far as statistical generalizability
is concerned, we find unspoken pitfalls and caveats with this approach. We
analytically show that their estimator is biased and uses error-prone
assumptions. We find that the estimator favors negative errors and yields poor
bootstrapped confidence intervals. We derive an unbiased alternative and
bolster our claims with empirical evidence from statistical simulation. Our
codebase is at http://github.com/castorini/meanmax.
