  One principled approach for provably efficient exploration is incorporating
the upper confidence bound (UCB) into the value function as a bonus. However,
UCB is specified to deal with linear and tabular settings and is incompatible
with Deep Reinforcement Learning (DRL). In this paper, we propose a principled
exploration method for DRL through Optimistic Bootstrapping and Backward
Induction (OB2I). OB2I constructs a general-purpose UCB-bonus through
non-parametric bootstrap in DRL. The UCB-bonus estimates the epistemic
uncertainty of state-action pairs for optimistic exploration. We build
theoretical connections between the proposed UCB-bonus and the LSVI-UCB in a
linear setting. We propagate future uncertainty in a time-consistent manner
through episodic backward update, which exploits the theoretical advantage and
empirically improves the sample-efficiency. Our experiments in the MNIST maze
and Atari suite suggest that OB2I outperforms several state-of-the-art
exploration approaches.
