  In order to learn universal sentence representations, previous methods focus
on complex recurrent neural networks or supervised learning. In this paper, we
propose a mean-max attention autoencoder (mean-max AAE) within the
encoder-decoder framework. Our autoencoder rely entirely on the MultiHead
self-attention mechanism to reconstruct the input sequence. In the encoding we
propose a mean-max strategy that applies both mean and max pooling operations
over the hidden vectors to capture diverse information of the input. To enable
the information to steer the reconstruction process dynamically, the decoder
performs attention over the mean-max representation. By training our model on a
large collection of unlabelled data, we obtain high-quality representations of
sentences. Experimental results on a broad range of 10 transfer tasks
demonstrate that our model outperforms the state-of-the-art unsupervised single
methods, including the classical skip-thoughts and the advanced
skip-thoughts+LN model. Furthermore, compared with the traditional recurrent
neural network, our mean-max AAE greatly reduce the training time.
