  Multiple-choice questions (MCQs) are commonly used across all levels of math
education since they can be deployed and graded at a large scale. A critical
component of MCQs is the distractors, i.e., incorrect answers crafted to
reflect student errors or misconceptions. Automatically generating them in math
MCQs, e.g., with large language models, has been challenging. In this work, we
propose a novel method to enhance the quality of generated distractors through
overgenerate-and-rank, training a ranking model to predict how likely
distractors are to be selected by real students. Experimental results on a
real-world dataset and human evaluation with math teachers show that our
ranking model increases alignment with human-authored distractors, although
human-authored ones are still preferred over generated ones.
