  Multiple-choice questions with item-writing flaws can negatively impact
student learning and skew analytics. These flaws are often present in
student-generated questions, making it difficult to assess their quality and
suitability for classroom usage. Existing methods for evaluating
multiple-choice questions often focus on machine readability metrics, without
considering their intended use within course materials and their pedagogical
implications. In this study, we compared the performance of a rule-based method
we developed to a machine-learning based method utilizing GPT-4 for the task of
automatically assessing multiple-choice questions based on 19 common
item-writing flaws. By analyzing 200 student-generated questions from four
different subject areas, we found that the rule-based method correctly detected
91% of the flaws identified by human annotators, as compared to 79% by GPT-4.
We demonstrated the effectiveness of the two methods in identifying common
item-writing flaws present in the student-generated questions across different
subject areas. The rule-based method can accurately and efficiently evaluate
multiple-choice questions from multiple domains, outperforming GPT-4 and going
beyond existing metrics that do not account for the educational use of such
questions. Finally, we discuss the potential for using these automated methods
to improve the quality of questions based on the identified flaws.
