  Offline Reinforcement Learning (RL) is a promising approach for learning
optimal policies in environments where direct exploration is expensive or
unfeasible. However, the adoption of such policies in practice is often
challenging, as they are hard to interpret within the application context, and
lack measures of uncertainty for the learned policy value and its decisions. To
overcome these issues, we propose an Expert-Supervised RL (ESRL) framework
which uses uncertainty quantification for offline policy learning. In
particular, we have three contributions: 1) the method can learn safe and
optimal policies through hypothesis testing, 2) ESRL allows for different
levels of risk averse implementations tailored to the application context, and
finally, 3) we propose a way to interpret ESRL's policy at every state through
posterior distributions, and use this framework to compute off-policy value
function posteriors. We provide theoretical guarantees for our estimators and
regret bounds consistent with Posterior Sampling for RL (PSRL). Sample
efficiency of ESRL is independent of the chosen risk aversion threshold and
quality of the behavior policy.
