  Encoding long sequences in Natural Language Processing (NLP) is a challenging
problem. Though recent pretraining language models achieve satisfying
performances in many NLP tasks, they are still restricted by a pre-defined
maximum length, making them challenging to be extended to longer sequences. So
some recent works utilize hierarchies to model long sequences. However, most of
them apply sequential models for upper hierarchies, suffering from long
dependency issues. In this paper, we alleviate these issues through a
graph-based method. We first chunk the sequence with a fixed length to model
the sentence-level information. We then leverage graphs to model intra- and
cross-sentence correlations with a new attention mechanism. Additionally, due
to limited standard benchmarks for long document classification (LDC), we
propose a new challenging benchmark, totaling six datasets with up to 53k
samples and 4034 average tokens' length. Evaluation shows our model surpasses
competitive baselines by 2.6% in F1 score, and 4.8% on the longest sequence
dataset. Our method is shown to outperform hierarchical sequential models with
better performance and scalability, especially for longer sequences.
