  We introduce a novel training principle for probabilistic models that is an
alternative to maximum likelihood. The proposed Generative Stochastic Networks
(GSN) framework is based on learning the transition operator of a Markov chain
whose stationary distribution estimates the data distribution. Because the
transition distribution is a conditional distribution generally involving a
small move, it has fewer dominant modes, being unimodal in the limit of small
moves. Thus, it is easier to learn, more like learning to perform supervised
function approximation, with gradients that can be obtained by
back-propagation. The theorems provided here generalize recent work on the
probabilistic interpretation of denoising auto-encoders and provide an
interesting justification for dependency networks and generalized
pseudolikelihood (along with defining an appropriate joint distribution and
sampling mechanism, even when the conditionals are not consistent). We study
how GSNs can be used with missing inputs and can be used to sample subsets of
variables given the rest. Successful experiments are conducted, validating
these theoretical results, on two image datasets and with a particular
architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows
training to proceed with backprop, without the need for layerwise pretraining.
