  Composing poetry or lyrics involves several creative factors, but a
challenging aspect of generation is the adherence to a more or less strict
metric and rhyming pattern. To address this challenge specifically, previous
work on the task has mainly focused on reverse language modeling, which brings
the critical selection of each rhyming word to the forefront of each verse. On
the other hand, reversing the word order requires that models be trained from
scratch with this task-specific goal and cannot take advantage of transfer
learning from a Pretrained Language Model (PLM). We propose a novel fine-tuning
approach that prepends the rhyming word at the start of each lyric, which
allows the critical rhyming decision to be made before the model commits to the
content of the lyric (as during reverse language modeling), but maintains
compatibility with the word order of regular PLMs as the lyric itself is still
generated in left-to-right order. We conducted extensive experiments to compare
this fine-tuning against the current state-of-the-art strategies for rhyming,
finding that our approach generates more readable text and better rhyming
capabilities. Furthermore, we furnish a high-quality dataset in English and 12
other languages, analyse the approach's feasibility in a multilingual context,
provide extensive experimental results shedding light on good and bad practices
for lyrics generation, and propose metrics to compare methods in the future.
