  Existing deep convolutional neural networks (CNNs) have shown their great
success on image classification. CNNs mainly consist of convolutional and
pooling layers, both of which are performed on local image areas without
considering the dependencies among different image regions. However, such
dependencies are very important for generating explicit image representation.
In contrast, recurrent neural networks (RNNs) are well known for their ability
of encoding contextual information among sequential data, and they only require
a limited number of network parameters. General RNNs can hardly be directly
applied on non-sequential data. Thus, we proposed the hierarchical RNNs
(HRNNs). In HRNNs, each RNN layer focuses on modeling spatial dependencies
among image regions from the same scale but different locations. While the
cross RNN scale connections target on modeling scale dependencies among regions
from the same location but different scales. Specifically, we propose two
recurrent neural network models: 1) hierarchical simple recurrent network
(HSRN), which is fast and has low computational cost; and 2) hierarchical
long-short term memory recurrent network (HLSTM), which performs better than
HSRN with the price of more computational cost.
  In this manuscript, we integrate CNNs with HRNNs, and develop end-to-end
convolutional hierarchical recurrent neural networks (C-HRNNs). C-HRNNs not
only make use of the representation power of CNNs, but also efficiently encodes
spatial and scale dependencies among different image regions. On four of the
most challenging object/scene image classification benchmarks, our C-HRNNs
achieve state-of-the-art results on Places 205, SUN 397, MIT indoor, and
competitive results on ILSVRC 2012.
