  The structure of the majority of modern deep neural networks is characterized
by uni- directional feed-forward connectivity across a very large number of
layers. By contrast, the architecture of the cortex of vertebrates contains
fewer hierarchical levels but many recurrent and feedback connections. Here we
show that a small, few-layer artificial neural network that employs feedback
will reach top level performance on a standard benchmark task, otherwise only
obtained by large feed-forward structures. To achieve this we use feed-forward
transfer entropy between neurons to structure feedback connectivity. Transfer
entropy can here intuitively be understood as a measure for the relevance of
certain pathways in the network, which are then amplified by feedback. Feedback
may therefore be key for high network performance in small brain-like
architectures.
