  Large language models (LLMs) have shown promising results in a wide array of
generative NLP tasks, such as summarization and machine translation. In the
context of narrative generation, however, existing models still do not capture
factors that contribute to producing consistent text. For instance, it is
logical that a piece of text or a story should be uniformly readable throughout
and that this form of complexity should be controllable. As such, if the
complexity of an input text prompt is rated first-grade reading level in the
Flesch Reading Ease test, then the generated text continuing the plot should
also be within this range of complexity. With this in mind, we introduce
Uniform Complexity for Text Generation (UCTG), a new benchmark test which
raises the challenge of making generative models observe uniform linguistic
properties with respect to prompts. We experiment with over 150+ linguistically
and cognitively motivated features for evaluating text complexity in humans and
generative models. From our results, we find that models such as GPT-2 struggle
to preserve the complexity of input prompts used in its generations, even if
finetuned with professionally written texts.
