  Classification of sequence data is the topic of interest for dynamic Bayesian
models and Recurrent Neural Networks (RNNs). While the former can explicitly
model the temporal dependencies between class variables, the latter have a
capability of learning representations. Several attempts have been made to
improve performance by combining these two approaches or increasing the
processing capability of the hidden units in RNNs. This often results in
complex models with a large number of learning parameters. In this paper, a
compact model is proposed which offers both representation learning and
temporal inference of class variables by rolling Restricted Boltzmann Machines
(RBMs) and class variables over time. We address the key issue of
intractability in this variant of RBMs by optimising a conditional
distribution, instead of a joint distribution. Experiments reported in the
paper on melody modelling and optical character recognition show that the
proposed model can outperform the state-of-the-art. Also, the experimental
results on optical character recognition, part-of-speech tagging and text
chunking demonstrate that our model is comparable to recurrent neural networks
with complex memory gates while requiring far fewer parameters.
