  The study of deep recurrent neural networks (RNNs) and, in particular, of
deep Reservoir Computing (RC) is gaining an increasing research attention in
the neural networks community. The recently introduced Deep Echo State Network
(DeepESN) model opened the way to an extremely efficient approach for designing
deep neural networks for temporal data. At the same time, the study of DeepESNs
allowed to shed light on the intrinsic properties of state dynamics developed
by hierarchical compositions of recurrent layers, i.e. on the bias of depth in
RNNs architectural design. In this paper, we summarize the advancements in the
development, analysis and applications of DeepESNs.
