  We present, to our knowledge, the first application of BERT to document
classification. A few characteristics of the task might lead one to think that
BERT is not the most appropriate model: syntactic structures matter less for
content categories, documents can often be longer than typical BERT input, and
documents often have multiple labels. Nevertheless, we show that a
straightforward classification model using BERT is able to achieve the state of
the art across four popular datasets. To address the computational expense
associated with BERT inference, we distill knowledge from BERT-large to small
bidirectional LSTMs, reaching BERT-base parity on multiple datasets using 30x
fewer parameters. The primary contribution of our paper is improved baselines
that can provide the foundation for future work.
