  In this paper, we study the task of improving the cohesion and coherence of
long-form text generated by language models. To this end, we propose RSTGen, a
framework that utilises Rhetorical Structure Theory (RST), a classical language
theory, to control the discourse structure, semantics and topics of generated
text. Firstly, we demonstrate our model's ability to control structural
discourse and semantic features of generated text in open generation
evaluation. Then we experiment on the two challenging long-form text tasks of
argument generation and story generation. Evaluation using automated metrics
and a metric with high correlation to human evaluation, shows that our model
performs competitively against existing models, while offering significantly
more controls over generated text than alternative methods.
