  By training to predict the next token in an unlabeled corpus, large language
models learn to perform many tasks without any labeled data. However, their
next-token-prediction objective arguably limits their performance in scenarios
that require planning, such as writing a coherent article. In this paper, we
train a module for planning the future writing process via a self-supervised
learning objective. Given the textual context, this planning module learns to
predict future abstract writing actions, which correspond to centroids in a
clustered text embedding space. By conditioning on these actions, our model
extends the successful language model formula to more abstract planning in an
unsupervised way. Empirically, we demonstrate that our method improves language
modeling performance in general, particularly with respect to the text
structure. Because our framework uses a planner module that is unsupervised and
external to the language model, new planner modules can be trained at large
scale and easily be shared with the community.
