  Class-based language models (LMs) have been long devised to address context
sparsity in $n$-gram LMs. In this study, we revisit this approach in the
context of neural LMs. We hypothesize that class-based prediction leads to an
implicit context aggregation for similar words and thus can improve
generalization for rare words. We map words that have a common WordNet hypernym
to the same class and train large neural LMs by gradually annealing from
predicting the class to token prediction during training. Empirically, this
curriculum learning strategy consistently improves perplexity over various
large, highly-performant state-of-the-art Transformer-based models on two
datasets, WikiText-103 and Arxiv. Our analysis shows that the performance
improvement is achieved without sacrificing performance on rare words. Finally,
we document other attempts that failed to yield empirical gains, and discuss
future directions for the adoption of class-based LMs on a larger scale.
