  This study investigates the efficacy of large language models (LLMs) as tools
for grading master-level student essays. Utilizing a sample of 60 essays in
political science, the study compares the accuracy of grades suggested by the
GPT-4 model with those awarded by university teachers. Results indicate that
while GPT-4 aligns with human grading standards on mean scores, it exhibits a
risk-averse grading pattern and its interrater reliability with human raters is
low. Furthermore, modifications in the grading instructions (prompt
engineering) do not significantly alter AI performance, suggesting that GPT-4
primarily assesses generic essay characteristics such as language quality
rather than adapting to nuanced grading criteria. These findings contribute to
the understanding of AI's potential and limitations in higher education,
highlighting the need for further development to enhance its adaptability and
sensitivity to specific educational assessment requirements.
