  Large language models (LLMs) fine-tuned by reinforcement learning from human
feedback (RLHF) are becoming more widely deployed. We coin the term
$\textit{Implicit Reward Model}$ (IRM) to refer to the changes that occur to an
LLM during RLHF that result in high-reward generations. We interpret IRMs, and
measure their divergence from the RLHF reward model used in the fine-tuning
process that induced them. By fitting a linear function to an LLM's IRM, a
reward model with the same type signature as the RLHF reward model is
constructed, allowing for direct comparison. Additionally, we validate our
construction of the IRM through cross-comparison with classifications of
features generated by an LLM based on their relevance to the RLHF reward model.
Better comprehending IRMs can help minimize discrepencies between LLM behavior
and training objectives, which we believe to be an essential component of the
$\textit{safety}$ and $\textit{alignment}$ of LLMs.
