  Generating complex behaviors that satisfy the preferences of non-expert users
is a crucial requirement for AI agents. Interactive reward learning from
trajectory comparisons (a.k.a. RLHF) is one way to allow non-expert users to
convey complex objectives by expressing preferences over short clips of agent
behaviors. Even though this parametric method can encode complex tacit
knowledge present in the underlying tasks, it implicitly assumes that the human
is unable to provide richer feedback than binary preference labels, leading to
intolerably high feedback complexity and poor user experience. While providing
a detailed symbolic closed-form specification of the objectives might be
tempting, it is not always feasible even for an expert user. However, in most
cases, humans are aware of how the agent should change its behavior along
meaningful axes to fulfill their underlying purpose, even if they are not able
to fully specify task objectives symbolically. Using this as motivation, we
introduce the notion of Relative Behavioral Attributes, which allows the users
to tweak the agent behavior through symbolic concepts (e.g., increasing the
softness or speed of agents' movement). We propose two practical methods that
can learn to model any kind of behavioral attributes from ordered behavior
clips. We demonstrate the effectiveness of our methods on four tasks with nine
different behavioral attributes, showing that once the attributes are learned,
end users can produce desirable agent behaviors relatively effortlessly, by
providing feedback just around ten times. This is over an order of magnitude
less than that required by the popular learning-from-human-preferences
baselines. The supplementary video and source code are available at:
https://guansuns.github.io/pages/rba.
