  Deep Reinforcement Learning (RL) is well known for being highly sensitive to
hyperparameters, requiring practitioners substantial efforts to optimize them
for the problem at hand. In recent years, the field of automated Reinforcement
Learning (AutoRL) has grown in popularity by trying to address this issue.
However, these approaches typically hinge on additional samples to select
well-performing hyperparameters, hindering sample-efficiency and practicality
in RL. Furthermore, most AutoRL methods are heavily based on already existing
AutoML methods, which were originally developed neglecting the additional
challenges inherent to RL due to its non-stationarities. In this work, we
propose a new approach for AutoRL, called Adaptive $Q$-Network (AdaQN), that is
tailored to RL to take into account the non-stationarity of the optimization
procedure without requiring additional samples. AdaQN learns several
$Q$-functions, each one trained with different hyperparameters, which are
updated online using the $Q$-function with the smallest approximation error as
a shared target. Our selection scheme simultaneously handles different
hyperparameters while coping with the non-stationarity induced by the RL
optimization procedure and being orthogonal to any critic-based RL algorithm.
We demonstrate that AdaQN is theoretically sound and empirically validate it in
MuJoCo control problems, showing benefits in sample-efficiency, overall
performance, training stability, and robustness to stochasticity.
