  Newly-introduced deep learning architectures, namely BERT, XLNet, RoBERTa and
ALBERT, have been proved to be robust on several NLP tasks. However, the
datasets trained on these architectures are fixed in terms of size and
generalizability. To relieve this issue, we apply one of the most inexpensive
solutions to update these datasets. We call this approach BET by which we
analyze the backtranslation data augmentation on the transformer-based
architectures. Using the Google Translate API with ten intermediary languages
from ten different language families, we externally evaluate the results in the
context of automatic paraphrase identification in a transformer-based
framework. Our findings suggest that BET improves the paraphrase identification
performance on the Microsoft Research Paraphrase Corpus (MRPC) to more than 3%
on both accuracy and F1 score. We also analyze the augmentation in the low-data
regime with downsampled versions of MRPC, Twitter Paraphrase Corpus (TPC) and
Quora Question Pairs. In many low-data cases, we observe a switch from a
failing model on the test set to reasonable performances. The results
demonstrate that BET is a highly promising data augmentation technique: to push
the current state-of-the-art of existing datasets and to bootstrap the
utilization of deep learning architectures in the low-data regime of a hundred
samples.
