  Written answers to open-ended questions can have a higher long-term effect on
learning than multiple-choice questions. However, it is critical that teachers
immediately review the answers, and ask to redo those that are incoherent. This
can be a difficult task and can be time-consuming for teachers. A possible
solution is to automate the detection of incoherent answers. One option is to
automate the review with Large Language Models (LLM). In this paper, we analyze
the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM,
and YOU. We used them with zero, one, two, three and four shots. We compared
their performance with the results of various classifiers trained with Machine
Learning (ML). We found that LLMs perform worse than MLs in detecting
incoherent answers. The difficulty seems to reside in recursive questions that
contain both questions and answers, and in responses from students with typical
fourth-grader misspellings. Upon closer examination, we have found that the
ChatGPT model faces the same challenges.
