  We show that deep narrow Boltzmann machines are universal approximators of
probability distributions on the activities of their visible units, provided
they have sufficiently many hidden layers, each containing the same number of
units as the visible layer. We show that, within certain parameter domains,
deep Boltzmann machines can be studied as feedforward networks. We provide
upper and lower bounds on the sufficient depth and width of universal
approximators. These results settle various intuitions regarding undirected
networks and, in particular, they show that deep narrow Boltzmann machines are
at least as compact universal approximators as narrow sigmoid belief networks
and restricted Boltzmann machines, with respect to the currently available
bounds for those models.
