  Modern language models can generate high-quality short texts. However, they
often meander or are incoherent when generating longer texts. These issues
arise from the next-token-only language modeling objective. Recent work in
self-supervised learning suggests that models can learn good latent
representations via contrastive learning, which can be effective for
discriminative tasks. Our work analyzes the application of contrastive
representations for generative tasks, like long text generation. We propose one
approach for leveraging constrastive representations, which we call Time
Control (TC). TC first learns a contrastive representation of the target text
domain, then generates text by decoding from these representations. Compared to
domain-specific methods and fine-tuning GPT2 across a variety of text domains,
TC performs competitively to methods specific for learning sentence
representations on discourse coherence. On long text generation settings, TC
preserves the text structure both in terms of ordering (up to $+15\%$ better)
and text length consistency (up to $+90\%$ better).
