  Automated Essay Scoring automates the grading process of essays, providing a
great advantage for improving the writing proficiency of students. While
holistic essay scoring research is prevalent, a noticeable gap exists in
scoring essays for specific quality traits. In this work, we focus on the
relevance trait, which measures the ability of the student to stay on-topic
throughout the entire essay. We propose a novel approach for graded relevance
scoring of written essays that employs dense retrieval encoders. Dense
representations of essays at different relevance levels then form clusters in
the embeddings space, such that their centroids are potentially separate enough
to effectively represent their relevance levels. We hence use the simple
1-Nearest-Neighbor classification over those centroids to determine the
relevance level of an unseen essay. As an effective unsupervised dense encoder,
we leverage Contriever, which is pre-trained with contrastive learning and
demonstrated comparable performance to supervised dense retrieval models. We
tested our approach on both task-specific (i.e., training and testing on same
task) and cross-task (i.e., testing on unseen task) scenarios using the widely
used ASAP++ dataset. Our method establishes a new state-of-the-art performance
in the task-specific scenario, while its extension for the cross-task scenario
exhibited a performance that is on par with the state-of-the-art model for that
scenario. We also analyzed the performance of our approach in a more practical
few-shot scenario, showing that it can significantly reduce the labeling cost
while sacrificing only 10% of its effectiveness.
