  Despite the success of text-to-text pre-trained models in various natural
language generation (NLG) tasks, the generation performance is largely
restricted by the number of labeled data in downstream tasks, particularly in
data-to-text generation tasks. Existing works mostly utilize abundant unlabeled
structured data to conduct unsupervised pre-training for task adaption, which
fail to model the complex relationship between source structured data and
target texts. Thus, we introduce self-training as a better few-shot learner
than task-adaptive pre-training, which explicitly captures this relationship
via pseudo-labeled data generated by the pre-trained model. To alleviate the
side-effect of low-quality pseudo-labeled data during self-training, we propose
a novel method called Curriculum-Based Self-Training (CBST) to effectively
leverage unlabeled data in a rearranged order determined by the difficulty of
text generation. Experimental results show that our method can outperform
fine-tuning and task-adaptive pre-training methods, and achieve
state-of-the-art performance in the few-shot setting of data-to-text
generation.
