  Learning from human feedback has shown to be a useful approach in acquiring
robot reward functions. However, expert feedback is often assumed to be drawn
from an underlying unimodal reward function. This assumption does not always
hold including in settings where multiple experts provide data or when a single
expert provides data for different tasks -- we thus go beyond learning a
unimodal reward and focus on learning a multimodal reward function. We
formulate the multimodal reward learning as a mixture learning problem and
develop a novel ranking-based learning approach, where the experts are only
required to rank a given set of trajectories. Furthermore, as access to
interaction data is often expensive in robotics, we develop an active querying
approach to accelerate the learning process. We conduct experiments and user
studies using a multi-task variant of OpenAI's LunarLander and a real Fetch
robot, where we collect data from multiple users with different preferences.
The results suggest that our approach can efficiently learn multimodal reward
functions, and improve data-efficiency over benchmark methods that we adapt to
our learning problem.
