  We introduce a new area of study in the field of educational Natural Language
Processing: Automated Long Answer Grading (ALAG). Distinguishing itself from
Automated Short Answer Grading (ASAG) and Automated Essay Grading (AEG), ALAG
presents unique challenges due to the complexity and multifaceted nature of
fact-based long answers. To study ALAG, we introduce RiceChem, a dataset
derived from a college chemistry course, featuring real student responses to
long-answer questions with an average word count notably higher than typical
ASAG datasets. We propose a novel approach to ALAG by formulating it as a
rubric entailment problem, employing natural language inference models to
verify whether each criterion, represented by a rubric item, is addressed in
the student's response. This formulation enables the effective use of MNLI for
transfer learning, significantly improving the performance of models on the
RiceChem dataset. We demonstrate the importance of rubric-based formulation in
ALAG, showcasing its superiority over traditional score-based approaches in
capturing the nuances of student responses. We also investigate the performance
of models in cold start scenarios, providing valuable insights into the
practical deployment considerations in educational settings. Lastly, we
benchmark state-of-the-art open-sourced Large Language Models (LLMs) on
RiceChem and compare their results to GPT models, highlighting the increased
complexity of ALAG compared to ASAG. Despite leveraging the benefits of a
rubric-based approach and transfer learning from MNLI, the lower performance of
LLMs on RiceChem underscores the significant difficulty posed by the ALAG task.
With this work, we offer a fresh perspective on grading long, fact-based
answers and introduce a new dataset to stimulate further research in this
important area. Code:
\url{https://github.com/luffycodes/Automated-Long-Answer-Grading}.
