  The semantics of a text is manifested not only by what is read, but also by
what is not read. In this article, we will study how the implicit "not read"
information such as end-of-paragraph (\eop) and end-of-sequence (\eos) affect
the quality of text generation. Specifically, we find that the pre-trained
language model GPT2 can generate better continuations by learning to generate
the \eop in the fine-tuning stage. Experimental results on English story
generation show that \eop can lead to higher BLEU score and lower \eos
perplexity. We also conduct experiments on a self-collected Chinese essay
dataset with Chinese-GPT2, a character level LM without \eop or \eos during
pre-training. Experimental results show that the Chinese GPT2 can generate
better essay endings with \eop.
