  Open-ended questions, which require students to produce multi-word,
nontrivial responses, are a popular tool for formative assessment as they
provide more specific insights into what students do and don't know. However,
grading open-ended questions can be time-consuming leading teachers to resort
to simpler question formats or conduct fewer formative assessments. While there
has been a longstanding interest in automating of short-answer grading (ASAG),
but previous approaches have been technically complex, limiting their use in
formative assessment contexts. The newest generation of Large Language Models
(LLMs) potentially makes grading short answer questions more feasible. This
paper investigates the potential for the newest version of LLMs to be used in
ASAG, specifically in the grading of short answer questions for formative
assessments, in two ways. First, it introduces a novel dataset of short answer
reading comprehension questions, drawn from a set of reading assessments
conducted with over 150 students in Ghana. This dataset allows for the
evaluation of LLMs in a new context, as they are predominantly designed and
trained on data from high-income North American countries. Second, the paper
empirically evaluates how well various configurations of generative LLMs grade
student short answer responses compared to expert human raters. The findings
show that GPT-4, with minimal prompt engineering, performed extremely well on
grading the novel dataset (QWK 0.92, F1 0.89), reaching near parity with expert
human raters. To our knowledge this work is the first to empirically evaluate
the performance of generative LLMs on short answer reading comprehension
questions using real student data, with low technical hurdles to attaining this
performance. These findings suggest that generative LLMs could be used to grade
formative literacy assessment tasks.
