  Reservoir computing is a machine learning framework where the readouts from a
nonlinear system (the reservoir) are trained so that the output from the
reservoir, when forced with an input signal, reproduces a desired output
signal. A common implementation of reservoir computers is to use a recurrent
neural network as the reservoir. The design of this network can have
significant effects on the performance of the reservoir computer. In this paper
we study the effect of the node activation function on the ability of reservoir
computers to learn and predict chaotic time series. We find that the Forecast
Horizon (FH), the time during which the reservoir's predictions remain
accurate, can vary by an order of magnitude across a set of 16 activation
functions used in machine learning. By using different functions from this set,
and by modifying their parameters, we explore whether the entropy of node
activation levels or the curvature of the activation functions determine the
predictive ability of the reservoirs. We find that the FH is low when the
activation function is used in a region where it has low curvature, and a
positive correlation between curvature and FH. For the activation functions
studied we find that the largest FH generally occurs at intermediate levels of
the entropy of node activation levels. Our results show that the performance of
reservoir computers is very sensitive to the activation function shape.
Therefore, modifying this shape in hyperparameter optimization algorithms can
lead to improvements in reservoir computer performance.
