  The use of large pretrained neural networks to create contextualized word
embeddings has drastically improved performance on several natural language
processing (NLP) tasks. These computationally expensive models have begun to be
applied to domain-specific NLP tasks such as re-hospitalization prediction from
clinical notes. This paper demonstrates that using large pretrained models
produces excellent results on common learning analytics tasks. Pre-training
deep language models using student forum data from a wide array of online
courses improves performance beyond the state of the art on three text
classification tasks. We also show that a smaller, distilled version of our
model produces the best results on two of the three tasks while limiting
computational cost. We make both models available to the research community at
large.
