  Human-in-the-loop reinforcement learning allows the training of agents
through various interfaces, even for non-expert humans. Recently,
preference-based methods (PbRL), where the human has to give his preference
over two trajectories, increased in popularity since they allow training in
domains where more direct feedback is hard to formulate. However, the current
PBRL methods have limitations and do not provide humans with an expressive
interface for giving feedback. With this work, we propose a new
preference-based learning method that provides humans with a more expressive
interface to provide their preference over trajectories and a factual
explanation (or annotation of why they have this preference). These
explanations allow the human to explain what parts of the trajectory are most
relevant for the preference. We allow the expression of the explanations over
individual trajectory steps. We evaluate our method in various simulations
using a simulated human oracle (with realistic restrictions), and our results
show that our extended feedback can improve the speed of learning.
