  In this paper, we study offline Reinforcement Learning with Human Feedback
(RLHF) where we aim to learn the human's underlying reward and the MDP's
optimal policy from a set of trajectories induced by human choices. RLHF is
challenging for multiple reasons: large state space but limited human feedback,
the bounded rationality of human decisions, and the off-policy distribution
shift. In this paper, we focus on the Dynamic Discrete Choice (DDC) model for
modeling and understanding human choices. DCC, rooted in econometrics and
decision theory, is widely used to model a human decision-making process with
forward-looking and bounded rationality. We propose a
\underline{D}ynamic-\underline{C}hoice-\underline{P}essimistic-\underline{P}olicy-\underline{O}ptimization
(DCPPO) method. \ The method involves a three-stage process: The first step is
to estimate the human behavior policy and the state-action value function via
maximum likelihood estimation (MLE); the second step recovers the human reward
function via minimizing Bellman mean squared error using the learned value
functions; the third step is to plug in the learned reward and invoke
pessimistic value iteration for finding a near-optimal policy. With only
single-policy coverage (i.e., optimal policy) of the dataset, we prove that the
suboptimality of DCPPO almost matches the classical pessimistic offline RL
algorithm in terms of suboptimality's dependency on distribution shift and
dimension. To the best of our knowledge, this paper presents the first
theoretical guarantees for off-policy offline RLHF with dynamic discrete choice
model.
