  Adversarial training has been shown effective at endowing the learned
representations with stronger generalization ability. However, it typically
requires expensive computation to determine the direction of the injected
perturbations. In this paper, we introduce a set of simple yet effective data
augmentation strategies dubbed cutoff, where part of the information within an
input sentence is erased to yield its restricted views (during the fine-tuning
stage). Notably, this process relies merely on stochastic sampling and thus
adds little computational overhead. A Jensen-Shannon Divergence consistency
loss is further utilized to incorporate these augmented samples into the
training objective in a principled manner. To verify the effectiveness of the
proposed strategies, we apply cutoff to both natural language understanding and
generation problems. On the GLUE benchmark, it is demonstrated that cutoff, in
spite of its simplicity, performs on par or better than several competitive
adversarial-based approaches. We further extend cutoff to machine translation
and observe significant gains in BLEU scores (based upon the Transformer Base
model). Moreover, cutoff consistently outperforms adversarial training and
achieves state-of-the-art results on the IWSLT2014 German-English dataset.
