  Language Models (LMs) often struggle with linguistic understanding at the
discourse level, even though discourse patterns such as coherence, cohesion,
and narrative flow are prevalent in their pre-training data. Current methods
address these challenges only after the pre-training phase, relying on
expensive human annotated data to align the model. To improve the discourse
capabilities of LMs already at the pre-training stage, we introduce DEPTH, an
encoder-decoder model that learns to represent sentences using a
discourse-oriented pre-training objective. DEPTH combines hierarchical sentence
representations with two objectives: (1) Sentence Un-Shuffling, and (2)
Span-Corruption. This approach trains the model to represent both
sub-word-level and sentence-level dependencies over a massive amount of
unstructured text. When trained either from scratch or continuing from a
pre-trained T5 checkpoint, DEPTH learns semantic and discourse-level
representations faster than T5, outperforming it in span-corruption loss
despite the additional sentence-un-shuffling objective. Evaluations on the
GLUE, DiscoEval, and NI benchmarks demonstrate DEPTH's ability to quickly learn
diverse downstream tasks, which require syntactic, semantic, and discourse
capabilities. Overall, our approach extends the discourse capabilities of T5,
while minimally impacting other natural language understanding (NLU)
capabilities in the resulting LM.
