  In this paper, we investigate improvements to the GEC sequence tagging
architecture with a focus on ensembling of recent cutting-edge
Transformer-based encoders in Large configurations. We encourage ensembling
models by majority votes on span-level edits because this approach is tolerant
to the model architecture and vocabulary size. Our best ensemble achieves a new
SOTA result with an $F_{0.5}$ score of 76.05 on BEA-2019 (test), even without
pre-training on synthetic datasets. In addition, we perform knowledge
distillation with a trained ensemble to generate new synthetic training
datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model
that is pretrained on the generated Troy-datasets in combination with the
publicly available synthetic PIE dataset achieves a near-SOTA (To the best of
our knowledge, our best single model gives way only to much heavier T5 model
result with an $F_{0.5}$ score of 73.21 on BEA-2019 (test). The code, datasets,
and trained models are publicly available).
