  Artificial neural networks (ANNs) are known to be powerful methods for many
hard problems (e.g. image classification, speech recognition or time series
prediction). However, these models tend to produce black-box results and are
often difficult to interpret. Layer-wise relevance propagation (LRP) is a
widely used technique to understand how ANN models come to their conclusion and
to understand what a model has learned. Here, we focus on Echo State Networks
(ESNs) as a certain type of recurrent neural networks, also known as reservoir
computing. ESNs are easy to train and only require a small number of trainable
parameters, but are still black-box models. We show how LRP can be applied to
ESNs in order to open the black-box. We also show how ESNs can be used not only
for time series prediction but also for image classification: Our ESN model
serves as a detector for El Nino Southern Oscillation (ENSO) from sea surface
temperature anomalies. ENSO is actually a well-known problem and has been
extensively discussed before. But here we use this simple problem to
demonstrate how LRP can significantly enhance the explainablility of ESNs.
