  Recurrent Neural Networks (RNNs) have demonstrated their outstanding ability
in sequence tasks and have achieved state-of-the-art in wide range of
applications, such as industrial, medical, economic and linguistic. Echo State
Network (ESN) is simple type of RNNs and has emerged in the last decade as an
alternative to gradient descent training based RNNs. ESN, with a strong
theoretical ground, is practical, conceptually simple, easy to implement. It
avoids non-converging and computationally expensive in the gradient descent
methods. Since ESN was put forward in 2002, abundant existing works have
promoted the progress of ESN, and the recently introduced Deep ESN model opened
the way to uniting the merits of deep learning and ESNs. Besides, the
combinations of ESNs with other machine learning models have also overperformed
baselines in some applications. However, the apparent simplicity of ESNs can
sometimes be deceptive and successfully applying ESNs needs some experience.
Thus, in this paper, we categorize the ESN-based methods to basic ESNs,
DeepESNs and combinations, then analyze them from the perspective of
theoretical studies, network designs and specific applications. Finally, we
discuss the challenges and opportunities of ESNs by summarizing the open
questions and proposing possible future works.
