  Over-fitting is a dreaded foe in challenge-based competitions. Because
participants rely on public leaderboards to evaluate and refine their models,
there is always the danger they might over-fit to the holdout data supporting
the leaderboard. The recently published Ladder algorithm aims to address this
problem by preventing the participants from exploiting willingly or
inadvertently minor fluctuations in public leaderboard scores during model
refinement. In this paper, we report a vulnerability of the Ladder that induces
severe over-fitting of the leaderboard when the sample size is small. To
circumvent this attack, we propose a variation of the Ladder that releases a
bootstrapped estimate of the public leaderboard score instead of providing
participants with a direct measure of performance. We also extend the scope of
the Ladder to arbitrary performance metrics by relying on a more broadly
applicable testing procedure based on the Bayesian bootstrap. Our method makes
it possible to use a leaderboard, with the technical and social advantages that
it provides, even in cases where data is scant.
