  Text classification algorithms investigate the intricate relationships
between words or phrases and attempt to deduce the document's interpretation.
In the last few years, these algorithms have progressed tremendously.
Transformer architecture and sentence encoders have proven to give superior
results on natural language processing tasks. But a major limitation of these
architectures is their applicability for text no longer than a few hundred
words. In this paper, we explore hierarchical transfer learning approaches for
long document classification. We employ pre-trained Universal Sentence Encoder
(USE) and Bidirectional Encoder Representations from Transformers (BERT) in a
hierarchical setup to capture better representations efficiently. Our proposed
models are conceptually simple where we divide the input data into chunks and
then pass this through base models of BERT and USE. Then output representation
for each chunk is then propagated through a shallow neural network comprising
of LSTMs or CNNs for classifying the text data. These extensions are evaluated
on 6 benchmark datasets. We show that USE + CNN/LSTM performs better than its
stand-alone baseline. Whereas the BERT + CNN/LSTM performs on par with its
stand-alone counterpart. However, the hierarchical BERT models are still
desirable as it avoids the quadratic complexity of the attention mechanism in
BERT. Along with the hierarchical approaches, this work also provides a
comparison of different deep learning algorithms like USE, BERT, HAN,
Longformer, and BigBird for long document classification. The Longformer
approach consistently performs well on most of the datasets.
