  It has been believed that stochastic feedforward neural networks (SFNNs) have
several advantages beyond deterministic deep neural networks (DNNs): they have
more expressive power allowing multi-modal mappings and regularize better due
to their stochastic nature. However, training large-scale SFNN is notoriously
harder. In this paper, we aim at developing efficient training methods for
SFNN, in particular using known architectures and pre-trained parameters of
DNN. To this end, we propose a new intermediate stochastic model, called
Simplified-SFNN, which can be built upon any baseline DNNand approximates
certain SFNN by simplifying its upper latent units above stochastic ones. The
main novelty of our approach is in establishing the connection between three
models, i.e., DNN->Simplified-SFNN->SFNN, which naturally leads to an efficient
training procedure of the stochastic models utilizing pre-trained parameters of
DNN. Using several popular DNNs, we show how they can be effectively
transferred to the corresponding stochastic models for both multi-modal and
classification tasks on MNIST, TFD, CASIA, CIFAR-10, CIFAR-100 and SVHN
datasets. In particular, we train a stochastic model of 28 layers and 36
million parameters, where training such a large-scale stochastic network is
significantly challenging without using Simplified-SFNN
