  Grammatical error detection (GED) in non-native writing requires systems to
identify a wide range of errors in text written by language learners. Error
detection as a purely supervised task can be challenging, as GED datasets are
limited in size and the label distributions are highly imbalanced.
Contextualized word representations offer a possible solution, as they can
efficiently capture compositional information in language and can be optimized
on large amounts of unsupervised data. In this paper, we perform a systematic
comparison of ELMo, BERT and Flair embeddings (Peters et al., 2017; Devlin et
al., 2018; Akbik et al., 2018) on a range of public GED datasets, and propose
an approach to effectively integrate such representations in current methods,
achieving a new state of the art on GED. We further analyze the strengths and
weaknesses of different contextual embeddings for the task at hand, and present
detailed analyses of their impact on different types of errors.
