  Sequence-to-sequence models based on LSTM and GRU are a most popular choice
for forecasting time series data reaching state-of-the-art performance.
Training such models can be delicate though. The two most common training
strategies within this context are teacher forcing (TF) and free running (FR).
TF can be used to help the model to converge faster but may provoke an exposure
bias issue due to a discrepancy between training and inference phase. FR helps
to avoid this but does not necessarily lead to better results, since it tends
to make the training slow and unstable instead. Scheduled sampling was the
first approach tackling these issues by picking the best from both worlds and
combining it into a curriculum learning (CL) strategy. Although scheduled
sampling seems to be a convincing alternative to FR and TF, we found that, even
if parametrized carefully, scheduled sampling may lead to premature termination
of the training when applied for time series forecasting. To mitigate the
problems of the above approaches we formalize CL strategies along the training
as well as the training iteration scale. We propose several new curricula, and
systematically evaluate their performance in two experimental sets. For our
experiments, we utilize six datasets generated from prominent chaotic systems.
We found that the newly proposed increasing training scale curricula with a
probabilistic iteration scale curriculum consistently outperforms previous
training strategies yielding an NRMSE improvement of up to 81% over FR or TF
training. For some datasets we additionally observe a reduced number of
training iterations. We observed that all models trained with the new curricula
yield higher prediction stability allowing for longer prediction horizons.
