  Offline reinforcement learning (RL) promises the ability to learn effective
policies solely using existing, static datasets, without any costly online
interaction. To do so, offline RL methods must handle distributional shift
between the dataset and the learned policy. The most common approach is to
learn conservative, or lower-bound, value functions, which underestimate the
return of out-of-distribution (OOD) actions. However, such methods exhibit one
notable drawback: policies optimized on such value functions can only behave
according to a fixed, possibly suboptimal, degree of conservatism. However,
this can be alleviated if we instead are able to learn policies for varying
degrees of conservatism at training time and devise a method to dynamically
choose one of them during evaluation. To do so, in this work, we propose
learning value functions that additionally condition on the degree of
conservatism, which we dub confidence-conditioned value functions. We derive a
new form of a Bellman backup that simultaneously learns Q-values for any degree
of confidence with high probability. By conditioning on confidence, our value
functions enable adaptive strategies during online evaluation by controlling
for confidence level using the history of observations thus far. This approach
can be implemented in practice by conditioning the Q-function from existing
conservative algorithms on the confidence.We theoretically show that our
learned value functions produce conservative estimates of the true value at any
desired confidence. Finally, we empirically show that our algorithm outperforms
existing conservative offline RL algorithms on multiple discrete control
domains.
