  Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL
agent learns to optimize a task using pair-wise preference-based feedback over
trajectories, rather than explicit reward signals. While PbRL has demonstrated
practical success in fine-tuning language models, existing theoretical work
focuses on regret minimization and fails to capture most of the practical
frameworks. In this study, we fill in such a gap between theoretical PbRL and
practical algorithms by proposing a theoretical reward-agnostic PbRL framework
where exploratory trajectories that enable accurate learning of hidden reward
functions are acquired before collecting any human feedback. Theoretical
analysis demonstrates that our algorithm requires less human feedback for
learning the optimal policy under preference-based models with linear
parameterization and unknown transitions, compared to the existing theoretical
literature. Specifically, our framework can incorporate linear and low-rank
MDPs with efficient sample complexity. Additionally, we investigate
reward-agnostic RL with action-based comparison feedback and introduce an
efficient querying algorithm tailored to this scenario.
