  The growth of scale and complexity of interactions between humans and robots
highlights the need for new computational methods to automatically evaluate
novel algorithms and applications. Exploring diverse scenarios of humans and
robots interacting in simulation can improve understanding of the robotic
system and avoid potentially costly failures in real-world settings. We
formulate this problem as a quality diversity (QD) problem, where the goal is
to discover diverse failure scenarios by simultaneously exploring both
environments and human actions. We focus on the shared autonomy domain, where
the robot attempts to infer the goal of a human operator, and adopt the QD
algorithm MAP-Elites to generate scenarios for two published algorithms in this
domain: shared autonomy via hindsight optimization and linear policy blending.
Some of the generated scenarios confirm previous theoretical findings, while
others are surprising and bring about a new understanding of state-of-the-art
implementations. Our experiments show that MAP-Elites outperforms Monte-Carlo
simulation and optimization based methods in effectively searching the scenario
space, highlighting its promise for automatic evaluation of algorithms in
human-robot interaction.
