  State-of-the-art poetry generation systems are often complex. They either
consist of task-specific model pipelines, incorporate prior knowledge in the
form of manually created constraints, or both. In contrast, end-to-end models
would not suffer from the overhead of having to model prior knowledge and could
learn the nuances of poetry from data alone, reducing the degree of human
supervision required. In this work, we investigate end-to-end poetry generation
conditioned on styles such as rhyme, meter, and alliteration. We identify and
address lack of training data and mismatching tokenization algorithms as
possible limitations of past attempts. In particular, we successfully pre-train
ByGPT5, a new token-free decoder-only language model, and fine-tune it on a
large custom corpus of English and German quatrains annotated with our styles.
We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and
ChatGPT, while also being more parameter efficient and performing favorably
compared to humans. In addition, we analyze its runtime performance and
demonstrate that it is not prone to memorization. We make our code, models, and
datasets publicly available.
