  Emergent behavior arising in a joint human-robot system cannot be fully
predicted based on an understanding of the individual agents. Typically, robot
behavior is governed by algorithms that optimize a reward function that should
quantitatively capture the joint system's goal. Although reward functions can
be updated to better match human needs, this is no guarantee that no
misalignment with the complex and variable human needs will occur. Algorithms
may learn undesirable behavior when interacting with the human and the
intrinsically unpredictable human-inhabited world, thereby producing further
misalignment with human users or bystanders. As a result, humans might behave
differently than anticipated, causing robots to learn differently and
undesirable behavior to emerge. With this short paper, we state that to design
for Human-Robot Interaction that mitigates such undesirable emergent behavior,
we need to complement advancements in human-robot interaction algorithms with
human factors knowledge and expertise. More specifically, we advocate a
three-pronged approach that we illustrate using a particularly challenging
example of safety-critical human-robot interaction: a driver interacting with a
semi-automated vehicle. Undesirable emergent behavior should be mitigated by a
combination of 1) including driver behavioral mechanisms in the vehicle's
algorithms and reward functions, 2) model-based approaches that account for
interaction-induced driver behavioral adaptations and 3) driver-centered
interaction design that promotes driver engagement with the semi-automated
vehicle, and the transparent communication of each agent's actions that allows
mutual support and adaptation. We provide examples from recent empirical work
in our group, in the hope this proves to be fruitful for discussing emergent
human-robot interaction.
