  It is widely believed that deep neural networks contain layer specialization,
wherein neural networks extract hierarchical features representing edges and
patterns in shallow layers and complete objects in deeper layers. Unlike common
feed-forward models that have distinct filters at each layer, recurrent
networks reuse the same parameters at various depths. In this work, we observe
that recurrent models exhibit the same hierarchical behaviors and the same
performance benefits with depth as feed-forward networks despite reusing the
same filters at every recurrence. By training models of various feed-forward
and recurrent architectures on several datasets for image classification as
well as maze solving, we show that recurrent networks have the ability to
closely emulate the behavior of non-recurrent deep models, often doing so with
far fewer parameters.
