  Reward modeling (a.k.a., preference modeling) is instrumental for aligning
large language models with human preferences, particularly within the context
of reinforcement learning from human feedback (RLHF). While conventional reward
models (RMs) have exhibited remarkable scalability, they oft struggle with
fundamental functionality such as arithmetic computation, code execution, and
factual lookup. In this paper, we propose a tool-augmented preference modeling
approach, named Themis, to address these limitations by empowering RMs with
access to external environments, including calculators and search engines. This
approach not only fosters synergy between tool utilization and reward grading
but also enhances interpretive capacity and scoring reliability. Our study
delves into the integration of external tools into RMs, enabling them to
interact with diverse external sources and construct task-specific tool
engagement and reasoning traces in an autoregressive manner. We validate our
approach across a wide range of domains, incorporating seven distinct external
tools. Our experimental results demonstrate a noteworthy overall improvement of
17.7% across eight tasks in preference ranking. Furthermore, our approach
outperforms Gopher 280B by 7.3% on TruthfulQA task in zero-shot evaluation. In
human evaluations, RLHF trained with Themis attains an average win rate of 32%
when compared to baselines across four distinct tasks. Additionally, we provide
a comprehensive collection of tool-related RM datasets, incorporating data from
seven distinct tool APIs, totaling 15,000 instances. We have made the code,
data, and model checkpoints publicly available to facilitate and inspire
further research
advancements\footnote{\url{https://github.com/ernie-research/Tool-Augmented-Reward-Model}}.
