  We introduce GENIUS: a conditional text generation model using sketches as
input, which can fill in the missing contexts for a given sketch (key
information consisting of textual spans, phrases, or words, concatenated by
mask tokens). GENIUS is pre-trained on a large-scale textual corpus with a
novel reconstruction from sketch objective using an extreme and selective
masking strategy, enabling it to generate diverse and high-quality texts given
sketches. Comparison with other competitive conditional language models (CLMs)
reveals the superiority of GENIUS's text generation quality. We further show
that GENIUS can be used as a strong and ready-to-use data augmentation tool for
various natural language processing (NLP) tasks. Most existing textual data
augmentation methods are either too conservative, by making small changes to
the original text, or too aggressive, by creating entirely new samples. With
GENIUS, we propose GeniusAug, which first extracts the target-aware sketches
from the original training set and then generates new samples based on the
sketches. Empirical experiments on 6 text classification datasets show that
GeniusAug significantly improves the models' performance in both
in-distribution (ID) and out-of-distribution (OOD) settings. We also
demonstrate the effectiveness of GeniusAug on named entity recognition (NER)
and machine reading comprehension (MRC) tasks. (Code and models are publicly
available at https://github.com/microsoft/SCGLab and
https://github.com/beyondguo/genius)
