  Lord's Paradox occurs when a continuous covariate is statistically controlled
for and the relationship between a continuous outcome and group status
indicator changes in both magnitude and direction. This phenomenon poses a
challenge to the notion of evidence-based policy, where data are supposed to be
self-evident. We examined 50 effect size estimates from 34 large-scale
educational interventions, and found that impact estimates are affected in
magnitude, with or without reversal in sign, when there is substantial baseline
imbalance. We also demonstrated that multilevel modelling can ameliorate the
divergence in sign and/or magnitude of effect estimation, which, together with
project specific knowledge, promises to help those who are presented with
conflicting or confusing evidence in decision making.
