  Neural text generation has made tremendous progress in various tasks. One
common characteristic of most of the tasks is that the texts are not restricted
to some rigid formats when generating. However, we may confront some special
text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi
(classical Chinese poetry of the Song dynasty), etc. The typical
characteristics of these texts are in three folds: (1) They must comply fully
with the rigid predefined formats. (2) They must obey some rhyming schemes. (3)
Although they are restricted to some formats, the sentence integrity must be
guaranteed. To the best of our knowledge, text generation based on the
predefined rigid formats has not been well investigated. Therefore, we propose
a simple and elegant framework named SongNet to tackle this problem. The
backbone of the framework is a Transformer-based auto-regressive language
model. Sets of symbols are tailor-designed to improve the modeling performance
especially on format, rhyme, and sentence integrity. We improve the attention
mechanism to impel the model to capture some future information on the format.
A pre-training and fine-tuning framework is designed to further improve the
generation quality. Extensive experiments conducted on two collected corpora
demonstrate that our proposed framework generates significantly better results
in terms of both automatic metrics and the human evaluation.
