  This work presents KoBigBird-large, a large size of Korean BigBird that
achieves state-of-the-art performance and allows long sequence processing for
Korean language understanding. Without further pretraining, we only transform
the architecture and extend the positional encoding with our proposed Tapered
Absolute Positional Encoding Representations (TAPER). In experiments,
KoBigBird-large shows state-of-the-art overall performance on Korean language
understanding benchmarks and the best performance on document classification
and question answering tasks for longer sequences against the competitive
baseline models. We publicly release our model here.
