  Early stopping based on the validation set performance is a popular approach
to find the right balance between under- and overfitting in the context of
supervised learning. However, in reinforcement learning, even for supervised
sub-problems such as world model learning, early stopping is not applicable as
the dataset is continually evolving. As a solution, we propose a new general
method that dynamically adjusts the update to data (UTD) ratio during training
based on under- and overfitting detection on a small subset of the continuously
collected experience not used for training. We apply our method to DreamerV2, a
state-of-the-art model-based reinforcement learning algorithm, and evaluate it
on the DeepMind Control Suite and the Atari $100$k benchmark. The results
demonstrate that one can better balance under- and overestimation by adjusting
the UTD ratio with our approach compared to the default setting in DreamerV2
and that it is competitive with an extensive hyperparameter search which is not
feasible for many applications. Our method eliminates the need to set the UTD
hyperparameter by hand and even leads to a higher robustness with regard to
other learning-related hyperparameters further reducing the amount of necessary
tuning.
