  Although several methods were proposed to address the problem of automated
essay scoring (AES) in the last 50 years, there is still much to desire in
terms of effectiveness. Large Language Models (LLMs) are transformer-based
models that demonstrate extraordinary capabilities on various tasks. In this
paper, we test the ability of LLMs, given their powerful linguistic knowledge,
to analyze and effectively score written essays. We experimented with two
popular LLMs, namely ChatGPT and Llama. We aim to check if these models can do
this task and, if so, how their performance is positioned among the
state-of-the-art (SOTA) models across two levels, holistically and per
individual writing trait. We utilized prompt-engineering tactics in designing
four different prompts to bring their maximum potential to this task. Our
experiments conducted on the ASAP dataset revealed several interesting
observations. First, choosing the right prompt depends highly on the model and
nature of the task. Second, the two LLMs exhibited comparable average
performance in AES, with a slight advantage for ChatGPT. Finally, despite the
performance gap between the two LLMs and SOTA models in terms of predictions,
they provide feedback to enhance the quality of the essays, which can
potentially help both teachers and students.
