  Instability and variability of Deep Reinforcement Learning (DRL) algorithms
tend to adversely affect their performance. Averaged-DQN is a simple extension
to the DQN algorithm, based on averaging previously learned Q-values estimates,
which leads to a more stable training procedure and improved performance by
reducing approximation error variance in the target values. To understand the
effect of the algorithm, we examine the source of value function estimation
errors and provide an analytical comparison within a simplified model. We
further present experiments on the Arcade Learning Environment benchmark that
demonstrate significantly improved stability and performance due to the
proposed extension.
