  Automated essay scoring (AES) is gaining increasing attention in the
education sector as it significantly reduces the burden of manual scoring and
allows ad hoc feedback for learners. Natural language processing based on
machine learning has been shown to be particularly suitable for text
classification and AES. While many machine-learning approaches for AES still
rely on a bag-of-words (BOW) approach, we consider a transformer-based approach
in this paper, compare its performance to a logistic regression model based on
the BOW approach and discuss their differences. The analysis is based on 2,088
email responses to a problem-solving task, that were manually labeled in terms
of politeness. Both transformer models considered in that analysis outperformed
without any hyper-parameter tuning the regression-based model. We argue that
for AES tasks such as politeness classification, the transformer-based approach
has significant advantages, while a BOW approach suffers from not taking word
order into account and reducing the words to their stem. Further, we show how
such models can help increase the accuracy of human raters, and we provide a
detailed instruction on how to implement transformer-based models for one's own
purpose.
