  We propose a new model for unsupervised document embedding. Leading existing
approaches either require complex inference or use recurrent neural networks
(RNN) that are difficult to parallelize. We take a different route and develop
a convolutional neural network (CNN) embedding model. Our CNN architecture is
fully parallelizable resulting in over 10x speedup in inference time over RNN
models. Parallelizable architecture enables to train deeper models where each
successive layer has increasingly larger receptive field and models longer
range semantic structure within the document. We additionally propose a fully
unsupervised learning algorithm to train this model based on stochastic forward
prediction. Empirical results on two public benchmarks show that our approach
produces comparable to state-of-the-art accuracy at a fraction of computational
cost.
