  We study the pre-train + fine-tune strategy for data-to-text tasks. Our
experiments indicate that text-to-text pre-training in the form of T5, enables
simple, end-to-end transformer based models to outperform pipelined neural
architectures tailored for data-to-text generation, as well as alternative
language model based pre-training techniques such as BERT and GPT-2.
Importantly, T5 pre-training leads to better generalization, as evidenced by
large improvements on out-of-domain test sets. We hope our work serves as a
useful baseline for future research, as transfer learning becomes ever more
prevalent for data-to-text tasks.
