  Building a good generative model for image has long been an important topic
in computer vision and machine learning. Restricted Boltzmann machine (RBM) is
one of such models that is simple but powerful. However, its restricted form
also has placed heavy constraints on the models representation power and
scalability. Many extensions have been invented based on RBM in order to
produce deeper architectures with greater power. The most famous ones among
them are deep belief network, which stacks multiple layer-wise pretrained RBMs
to form a hybrid model, and deep Boltzmann machine, which allows connections
between hidden units to form a multi-layer structure. In this paper, we present
a new method to compose RBMs to form a multi-layer network style architecture
and a training method that trains all layers jointly. We call the resulted
structure deep restricted Boltzmann network. We further explore the combination
of convolutional RBM with the normal fully connected RBM, which is made trivial
under our composition framework. Experiments show that our model can generate
descent images and outperform the normal RBM significantly in terms of image
quality and feature quality, without losing much efficiency for training.
