  The recent literature in text classification is biased towards short text
sequences (e.g., sentences or paragraphs). In real-world applications,
multi-page multi-paragraph documents are common and they cannot be efficiently
encoded by vanilla Transformer-based models. We compare different
Transformer-based Long Document Classification (TrLDC) approaches that aim to
mitigate the computational overhead of vanilla transformers to encode much
longer text, namely sparse attention and hierarchical encoding methods. We
examine several aspects of sparse attention (e.g., size of local attention
window, use of global attention) and hierarchical (e.g., document splitting
strategy) transformers on four document classification datasets covering
different domains. We observe a clear benefit from being able to process longer
text, and, based on our results, we derive practical advice of applying
Transformer-based models on long document classification tasks.
