  Designing reward functions is a challenging problem in AI and robotics.
Humans usually have a difficult time directly specifying all the desirable
behaviors that a robot needs to optimize. One common approach is to learn
reward functions from collected expert demonstrations. However, learning reward
functions from demonstrations introduces many challenges: some methods require
highly structured models, e.g. reward functions that are linear in some
predefined set of features, while others adopt less structured reward functions
that on the other hand require tremendous amount of data. In addition, humans
tend to have a difficult time providing demonstrations on robots with high
degrees of freedom, or even quantifying reward values for given demonstrations.
To address these challenges, we present a preference-based learning approach,
where as an alternative, the human feedback is only in the form of comparisons
between trajectories. Furthermore, we do not assume highly constrained
structures on the reward function. Instead, we model the reward function using
a Gaussian Process (GP) and propose a mathematical formulation to actively find
a GP using only human preferences. Our approach enables us to tackle both
inflexibility and data-inefficiency problems within a preference-based learning
framework. Our results in simulations and a user study suggest that our
approach can efficiently learn expressive reward functions for robotics tasks.
