  To automatically correct handwritten assignments, the traditional approach is
to use an OCR model to recognize characters and compare them to answers. The
OCR model easily gets confused on recognizing handwritten Chinese characters,
and the textual information of the answers is missing during the model
inference. However, teachers always have these answers in mind to review and
correct assignments. In this paper, we focus on the Chinese cloze tests
correction and propose a multimodal approach (named AiM). The encoded
representations of answers interact with the visual information of students'
handwriting. Instead of predicting 'right' or 'wrong', we perform the sequence
labeling on the answer text to infer which answer character differs from the
handwritten content in a fine-grained way. We take samples of OCR datasets as
the positive samples for this task, and develop a negative sample augmentation
method to scale up the training data. Experimental results show that AiM
outperforms OCR-based methods by a large margin. Extensive studies demonstrate
the effectiveness of our multimodal approach.
