  We consider the finite-horizon offline reinforcement learning (RL) setting,
and are motivated by the challenge of learning the policy at any step h in
dynamic programming (DP) algorithms. To learn this, it is sufficient to
evaluate the treatment effect of deviating from the behavioral policy at step h
after having optimized the policy for all future steps. Since the policy at any
step can affect next-state distributions, the related distributional shift
challenges can make this problem far more statistically hard than estimating
such treatment effects in the stochastic contextual bandit setting. However,
the hardness of many real-world RL instances lies between the two regimes. We
develop a flexible and general method called selective uncertainty propagation
for confidence interval construction that adapts to the hardness of the
associated distribution shift challenges. We show benefits of our approach on
toy environments and demonstrate the benefits of these techniques for offline
policy learning.
