  This paper focuses on the data augmentation for low-resource NLP tasks where
the training set is limited. The existing solutions either leverage
task-independent heuristic rules (e.g., Synonym Replacement) or fine-tune
general-purpose pre-trained language models (e.g., GPT2) using the limited
training instances to produce new synthetic data. Consequently, they have
trivial task-specific knowledge and are limited to yielding low-quality
synthetic data. To combat this issue, we propose Knowledge Mixture Data
Augmentation Model (KnowDA) which is an Seq2Seq language model pre-trained on a
mixture of diverse NLP tasks under a novel framework of Knowledge Mixture
Training (KoMT). The goal of KoMT is to condense diverse NLP task-specific
knowledge into the single KnowDA model (i.e., all-in-one) such that KnowDA
could utilize these knowledge to quickly grasp the inherent synthesis law of
the target task through limited training instances. Specifically, KoMT
reformulates input examples from various heterogeneous NLP tasks into a unified
text-to-text format, and employs denoising training objectives in different
granularity to learn to reconstruct partial or complete samples. To the best of
our knowledge, we are the first attempt to apply 100+ NLP multi-task training
for data augmentation. Extensive experiments show that i) the synthetic data
produced by KnowDA successfully improves performance of the strong pre-trained
language models (i.e., Bert, ALBert and Deberta) by a large margin on the
low-resource NLP benchmark FewGLUE, CoNLL'03 and WikiAnn; ii) KnowDA
successfully transfers the task knowledge to NLP tasks whose types are seen and
unseen in KoMT.
