  Reinforcement learning agents are faced with two types of uncertainty.
Epistemic uncertainty stems from limited data and is useful for exploration,
whereas aleatoric uncertainty arises from stochastic environments and must be
accounted for in risk-sensitive applications. We highlight the challenges
involved in simultaneously estimating both of them, and propose a framework for
disentangling and estimating these uncertainties on learned Q-values. We derive
unbiased estimators of these uncertainties and introduce an uncertainty-aware
DQN algorithm, which we show exhibits safe learning behavior and outperforms
other DQN variants on the MinAtar testbed.
