  Receiving timely and personalized feedback is essential for second-language
learners, especially when human instructors are unavailable. This study
explores the effectiveness of Large Language Models (LLMs), including both
proprietary and open-source models, for Automated Essay Scoring (AES). Through
extensive experiments with public and private datasets, we find that while LLMs
do not surpass conventional state-of-the-art (SOTA) grading models in
performance, they exhibit notable consistency, generalizability, and
explainability. We propose an open-source LLM-based AES system, inspired by the
dual-process theory. Our system offers accurate grading and high-quality
feedback, at least comparable to that of fine-tuned proprietary LLMs, in
addition to its ability to alleviate misgrading. Furthermore, we conduct
human-AI co-grading experiments with both novice and expert graders. We find
that our system not only automates the grading process but also enhances the
performance and efficiency of human graders, particularly for essays where the
model has lower confidence. These results highlight the potential of LLMs to
facilitate effective human-AI collaboration in the educational context,
potentially transforming learning experiences through AI-generated feedback.
