  Comparative Judgement is an assessment method where item ratings are
estimated based on rankings of subsets of the items. These rankings are
typically pairwise, with ratings taken to be the estimated parameters from
fitting a Bradley-Terry model. Likelihood penalization is often employed.
Adaptive scheduling of the comparisons can increase the efficiency of the
assessment. We show that the most commonly used penalty is not the
best-performing penalty under adaptive scheduling and can lead to substantial
bias in parameter estimates. We demonstrate this using simulated and real data
and provide a theoretical explanation for the relative performance of the
penalties considered. Further, we propose a superior approach based on
bootstrapping. It is shown to produce better parameter estimates for adaptive
schedules and to be robust to variations in underlying strength distributions
and initial penalization method.
