  In temporal-difference reinforcement learning algorithms, variance in value
estimation can cause instability and overestimation of the maximal target
value. Many algorithms have been proposed to reduce overestimation, including
several recent ensemble methods, however none have shown success in
sample-efficient learning through addressing estimation variance as the root
cause of overestimation. In this paper, we propose MeanQ, a simple ensemble
method that estimates target values as ensemble means. Despite its simplicity,
MeanQ shows remarkable sample efficiency in experiments on the Atari Learning
Environment benchmark. Importantly, we find that an ensemble of size 5
sufficiently reduces estimation variance to obviate the lagging target network,
eliminating it as a source of bias and further gaining sample efficiency. We
justify intuitively and empirically the design choices in MeanQ, including the
necessity of independent experience sampling. On a set of 26 benchmark Atari
environments, MeanQ outperforms all tested baselines, including the best
available baseline, SUNRISE, at 100K interaction steps in 16/26 environments,
and by 68% on average. MeanQ also outperforms Rainbow DQN at 500K steps in
21/26 environments, and by 49% on average, and achieves average human-level
performance using 200K ($\pm$100K) interaction steps. Our implementation is
available at https://github.com/indylab/MeanQ.
