  Residual connections have been proposed as an architecture-based inductive
bias to mitigate the problem of exploding and vanishing gradients and increased
task performance in both feed-forward and recurrent networks (RNNs) when
trained with the backpropagation algorithm. Yet, little is known about how
residual connections in RNNs influence their dynamics and fading memory
properties. Here, we introduce weakly coupled residual recurrent networks
(WCRNNs) in which residual connections result in well-defined Lyapunov
exponents and allow for studying properties of fading memory. We investigate
how the residual connections of WCRNNs influence their performance, network
dynamics, and memory properties on a set of benchmark tasks. We show that
several distinct forms of residual connections yield effective inductive biases
that result in increased network expressivity. In particular, those are
residual connections that (i) result in network dynamics at the proximity of
the edge of chaos, (ii) allow networks to capitalize on characteristic spectral
properties of the data, and (iii) result in heterogeneous memory properties. In
addition, we demonstrate how our results can be extended to non-linear
residuals and introduce a weakly coupled residual initialization scheme that
can be used for Elman RNNs.
