  In this work, we propose a novel cross Q-learning algorithm, aim at
alleviating the well-known overestimation problem in value-based reinforcement
learning methods, particularly in the deep Q-networks where the overestimation
is exaggerated by function approximation errors. Our algorithm builds on double
Q-learning, by maintaining a set of parallel models and estimate the Q-value
based on a randomly selected network, which leads to reduced overestimation
bias as well as the variance. We provide empirical evidence on the advantages
of our method by evaluating on some benchmark environment, the experimental
results demonstrate significant improvement of performance in reducing the
overestimation bias and stabilizing the training, further leading to better
derived policies.
