  In reinforcement learning, it is typical to use the empirically observed
transitions and rewards to estimate the value of a policy via either
model-based or Q-fitting approaches. Although straightforward, these techniques
in general yield biased estimates of the true value of the policy. In this
work, we investigate the potential for statistical bootstrapping to be used as
a way to take these biased estimates and produce calibrated confidence
intervals for the true value of the policy. We identify conditions -
specifically, sufficient data size and sufficient coverage - under which
statistical bootstrapping in this setting is guaranteed to yield correct
confidence intervals. In practical situations, these conditions often do not
hold, and so we discuss and propose mechanisms that can be employed to mitigate
their effects. We evaluate our proposed method and show that it can yield
accurate confidence intervals in a variety of conditions, including challenging
continuous control environments and small data regimes.
