  Deep reinforcement learning can learn effective policies for a wide range of
tasks, but is notoriously difficult to use due to instability and sensitivity
to hyperparameters. The reasons for this remain unclear. When using standard
supervised methods (e.g., for bandits), on-policy data collection provides
"hard negatives" that correct the model in precisely those states and actions
that the policy is likely to visit. We call this phenomenon "corrective
feedback." We show that bootstrapping-based Q-learning algorithms do not
necessarily benefit from this corrective feedback, and training on the
experience collected by the algorithm is not sufficient to correct errors in
the Q-function. In fact, Q-learning and related methods can exhibit
pathological interactions between the distribution of experience collected by
the agent and the policy induced by training on that experience, leading to
potential instability, sub-optimal convergence, and poor results when learning
from noisy, sparse or delayed rewards. We demonstrate the existence of this
problem, both theoretically and empirically. We then show that a specific
correction to the data distribution can mitigate this issue. Based on these
observations, we propose a new algorithm, DisCor, which computes an
approximation to this optimal distribution and uses it to re-weight the
transitions used for training, resulting in substantial improvements in a range
of challenging RL settings, such as multi-task learning and learning from noisy
reward signals. Blog post presenting a summary of this work is available at:
https://bair.berkeley.edu/blog/2020/03/16/discor/.
