  Even though BERT achieves successful performance improvements in various
supervised learning tasks, applying BERT for unsupervised tasks still holds a
limitation that it requires repetitive inference for computing contextual
language representations. To resolve the limitation, we propose a novel deep
bidirectional language model called Transformer-based Text Autoencoder (T-TA).
The T-TA computes contextual language representations without repetition and
has benefits of the deep bidirectional architecture like BERT. In run-time
experiments on CPU environments, the proposed T-TA performs over six times
faster than the BERT-based model in the reranking task and twelve times faster
in the semantic similarity task. Furthermore, the T-TA shows competitive or
even better accuracies than those of BERT on the above tasks.
