  Feed-forward convolutional neural networks (CNNs) are currently
state-of-the-art for object classification tasks such as ImageNet. Further,
they are quantitatively accurate models of temporally-averaged responses of
neurons in the primate brain's visual system. However, biological visual
systems have two ubiquitous architectural features not shared with typical
CNNs: local recurrence within cortical areas, and long-range feedback from
downstream areas to upstream areas. Here we explored the role of recurrence in
improving classification performance. We found that standard forms of
recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the
ImageNet task. In contrast, novel cells that incorporated two structural
features, bypassing and gating, were able to boost task accuracy substantially.
We extended these design principles in an automated search over thousands of
model architectures, which identified novel local recurrent cells and
long-range feedback connections useful for object recognition. Moreover, these
task-optimized ConvRNNs matched the dynamics of neural activity in the primate
visual system better than feedforward networks, suggesting a role for the
brain's recurrent connections in performing difficult visual behaviors.
