  Sequence-to-sequence (Seq2seq) models have played an important role in the
recent success of various natural language processing methods, such as machine
translation, text summarization, and speech recognition. However, current
Seq2seq models have trouble preserving global latent information from a long
sequence of words. Variational autoencoder (VAE) alleviates this problem by
learning a continuous semantic space of the input sentence. However, it does
not solve the problem completely. In this paper, we propose a new recurrent
neural network (RNN)-based Seq2seq model, RNN semantic variational autoencoder
(RNN--SVAE), to better capture the global latent information of a sequence of
words. To reflect the meaning of words in a sentence properly, without regard
to its position within the sentence, we construct a document information vector
using the attention information between the final state of the encoder and
every prior hidden state. Then, the mean and standard deviation of the
continuous semantic space are learned by using this vector to take advantage of
the variational method. By using the document information vector to find the
semantic space of the sentence, it becomes possible to better capture the
global latent feature of the sentence. Experimental results of three natural
language tasks (i.e., language modeling, missing word imputation, paraphrase
identification) confirm that the proposed RNN--SVAE yields higher performance
than two benchmark models.
