  There has been an increased interest in data generation approaches to
grammatical error correction (GEC) using pseudo data. However, these approaches
suffer from several issues that make them inconvenient for real-world
deployment including a demand for large amounts of training data. On the other
hand, some errors based on grammatical rules may not necessarily require a
large amount of data if GEC models can realize grammatical generalization. This
study explores to what extent GEC models generalize grammatical knowledge
required for correcting errors. We introduce an analysis method using synthetic
and real GEC datasets with controlled vocabularies to evaluate whether models
can generalize to unseen errors. We found that a current standard
Transformer-based GEC model fails to realize grammatical generalization even in
simple settings with limited vocabulary and syntax, suggesting that it lacks
the generalization ability required to correct errors from provided training
examples.
