  Reinforcement learning from human feedback (RLHF) is a powerful technique for
training agents to perform difficult-to-specify tasks. However, human feedback
can be noisy, particularly when human teachers lack relevant knowledge or
experience. Levels of expertise vary across teachers, and a given teacher may
have differing levels of expertise for different components of a task. RLHF
algorithms that learn from multiple teachers therefore face an expertise
problem: the reliability of a given piece of feedback depends both on the
teacher that it comes from and how specialized that teacher is on relevant
components of the task. Existing state-of-the-art RLHF algorithms assume that
all evaluations come from the same distribution, obscuring this inter- and
intra-human variance, and preventing them from accounting for or taking
advantage of variations in expertise. We formalize this problem, implement it
as an extension of an existing RLHF benchmark, evaluate the performance of a
state-of-the-art RLHF algorithm, and explore techniques to improve query and
teacher selection. Our key contribution is to demonstrate and characterize the
expertise problem, and to provide an open-source implementation for testing
future solutions.
