  It is common to evaluate the performance of a machine learning model by
measuring its predictive power on a test dataset. This approach favors
complicated models that can smoothly fit complex functions and generalize well
from training data points. Although essential components of intelligence, speed
and data efficiency of this learning process are rarely reported or compared
between different candidate models. In this paper, we introduce a benchmark of
increasingly difficult tasks together with a data efficiency metric to measure
how quickly machine learning models learn from training data. We compare the
learning speed of some established sequential supervised models, such as RNNs,
LSTMs, or Transformers, with relatively less known alternative models based on
reservoir computing. The proposed tasks require a wide range of computational
primitives, such as memory or the ability to compute Boolean functions, to be
effectively solved. Surprisingly, we observe that reservoir computing systems
that rely on dynamically evolving feature maps learn faster than fully
supervised methods trained with stochastic gradient optimization while
achieving comparable accuracy scores. The code, benchmark, trained models, and
results to reproduce our experiments are available at
https://github.com/hugcis/benchmark_learning_efficiency/ .
