  Convolutional Neural Network(CNN) has been widely used for image recognition
with great success. However, there are a number of limitations of the current
CNN based image recognition paradigm. First, the receptive field of CNN is
generally fixed, which limits its recognition capacity when the input image is
very large. Second, it lacks the computational scalability for dealing with
images with different sizes. Third, it is quite different from human visual
system for image recognition, which involves both feadforward and recurrent
proprocessing. This paper proposes a different paradigm of image recognition,
which can take advantages of variable scales of the input images, has more
computational scalabilities, and is more similar to image recognition by human
visual system. It is based on recurrent neural network (RNN) defined on image
scale with an embeded base CNN, which is named Scale Recurrent Neural
Network(SRNN). This RNN based approach makes it easier to deal with images with
variable sizes, and allows us to borrow existing RNN techniques, such as LSTM
and GRU, to further enhance the recognition accuracy. Our experiments show that
the recognition accuracy of a base CNN can be significantly boosted using the
proposed SRNN models. It also significantly outperforms the scale ensemble
method, which integrate the results of performing CNN to the input image at
different scales, although the computational overhead of using SRNN is
negligible.
