  We study the mixtures of factorizing probability distributions represented as
visible marginal distributions in stochastic layered networks. We take the
perspective of kernel transitions of distributions, which gives a unified
picture of distributed representations arising from Deep Belief Networks (DBN)
and other networks without lateral connections. We describe combinatorial and
geometric properties of the set of kernels and products of kernels realizable
by DBNs as the network parameters vary. We describe explicit classes of
probability distributions, including exponential families, that can be learned
by DBNs. We use these submodels to bound the maximal and the expected
Kullback-Leibler approximation errors of DBNs from above depending on the
number of hidden layers and units that they contain.
