  Preference-based reinforcement learning (PbRL) provides a natural way to
align RL agents' behavior with human desired outcomes, but is often restrained
by costly human feedback. To improve feedback efficiency, most existing PbRL
methods focus on selecting queries to maximally improve the overall quality of
the reward model, but counter-intuitively, we find that this may not
necessarily lead to improved performance. To unravel this mystery, we identify
a long-neglected issue in the query selection schemes of existing PbRL studies:
Query-Policy Misalignment. We show that the seemingly informative queries
selected to improve the overall quality of reward model actually may not align
with RL agents' interests, thus offering little help on policy learning and
eventually resulting in poor feedback efficiency. We show that this issue can
be effectively addressed via near on-policy query and a specially designed
hybrid experience replay, which together enforce the bidirectional query-policy
alignment. Simple yet elegant, our method can be easily incorporated into
existing approaches by changing only a few lines of code. We showcase in
comprehensive experiments that our method achieves substantial gains in both
human feedback and RL sample efficiency, demonstrating the importance of
addressing query-policy misalignment in PbRL tasks.
