  We seek to align agent behavior with a user's objectives in a reinforcement
learning setting with unknown dynamics, an unknown reward function, and unknown
unsafe states. The user knows the rewards and unsafe states, but querying the
user is expensive. To address this challenge, we propose an algorithm that
safely and interactively learns a model of the user's reward function. We start
with a generative model of initial states and a forward dynamics model trained
on off-policy data. Our method uses these models to synthesize hypothetical
behaviors, asks the user to label the behaviors with rewards, and trains a
neural network to predict the rewards. The key idea is to actively synthesize
the hypothetical behaviors from scratch by maximizing tractable proxies for the
value of information, without interacting with the environment. We call this
method reward query synthesis via trajectory optimization (ReQueST). We
evaluate ReQueST with simulated users on a state-based 2D navigation task and
the image-based Car Racing video game. The results show that ReQueST
significantly outperforms prior methods in learning reward models that transfer
to new environments with different initial state distributions. Moreover,
ReQueST safely trains the reward model to detect unsafe states, and corrects
reward hacking before deploying the agent.
