  Preference-based feedback is important for many applications in reinforcement
learning where direct evaluation of a reward function is not feasible. A
notable recent example arises in reinforcement learning from human feedback
(RLHF) on large language models. For many applications of RLHF, the cost of
acquiring the human feedback can be substantial. In this work, we take
advantage of the fact that one can often choose contexts at which to obtain
human feedback in order to most efficiently identify a good policy, and
formalize this as an offline contextual dueling bandit problem. We give an
upper-confidence-bound style algorithm for this problem and prove a polynomial
worst-case regret bound. We then provide empirical confirmation in a synthetic
setting that our approach outperforms existing methods. After, we extend the
setting and methodology for practical use in RLHF training of large language
models. Here, our method is able to reach better performance with fewer samples
of human preferences than multiple baselines on three real-world datasets.
