  Echo State Networks (ESNs) are time-series processing models working under
the Echo State Property (ESP) principle. The ESP is a notion of stability that
imposes an asymptotic fading of the memory of the input. On the other hand, the
resulting inherent architectural bias of ESNs may lead to an excessive loss of
information, which in turn harms the performance in certain tasks with long
short-term memory requirements. With the goal of bringing together the fading
memory property and the ability to retain as much memory as possible, in this
paper we introduce a new ESN architecture, called the Edge of Stability Echo
State Network (ES$^2$N). The introduced ES$^2$N model is based on defining the
reservoir layer as a convex combination of a nonlinear reservoir (as in the
standard ESN), and a linear reservoir that implements an orthogonal
transformation. We provide a thorough mathematical analysis of the introduced
model, proving that the whole eigenspectrum of the Jacobian of the ES$^2$N map
can be contained in an annular neighbourhood of a complex circle of
controllable radius, and exploit this property to demonstrate that the
ES$^2$N's forward dynamics evolves close to the edge-of-chaos regime by design.
Remarkably, our experimental analysis shows that the newly introduced reservoir
model is able to reach the theoretical maximum short-term memory capacity. At
the same time, in comparison to standard ESN, ES$^2$N is shown to offer an
excellent trade-off between memory and nonlinearity, as well as a significant
improvement of performance in autoregressive nonlinear modeling.
