  Effective peer assessment requires students to be attentive to the
deficiencies in the work they rate. Thus, their reviews should identify
problems. But what ways are there to check that they do? We attempt to automate
the process of deciding whether a review comment detects a problem. We use over
18,000 review comments that were labeled by the reviewees as either detecting
or not detecting a problem with the work. We deploy several traditional
machine-learning models, as well as neural-network models using GloVe and BERT
embeddings. We find that the best performer is the Hierarchical Attention
Network classifier, followed by the Bidirectional Gated Recurrent Units (GRU)
Attention and Capsule model with scores of 93.1% and 90.5% respectively. The
best non-neural network model was the support vector machine with a score of
89.71%. This is followed by the Stochastic Gradient Descent model and the
Logistic Regression model with 89.70% and 88.98%.
