  Imitation learning with human data has demonstrated remarkable success in
teaching robots in a wide range of skills. However, the inherent diversity in
human behavior leads to the emergence of multi-modal data distributions,
thereby presenting a formidable challenge for existing imitation learning
algorithms. Quantifying a model's capacity to capture and replicate this
diversity effectively is still an open problem. In this work, we introduce
simulation benchmark environments and the corresponding Datasets with Diverse
human Demonstrations for Imitation Learning (D3IL), designed explicitly to
evaluate a model's ability to learn multi-modal behavior. Our environments are
designed to involve multiple sub-tasks that need to be solved, consider
manipulation of multiple objects which increases the diversity of the behavior
and can only be solved by policies that rely on closed loop sensory feedback.
Other available datasets are missing at least one of these challenging
properties. To address the challenge of diversity quantification, we introduce
tractable metrics that provide valuable insights into a model's ability to
acquire and reproduce diverse behaviors. These metrics offer a practical means
to assess the robustness and versatility of imitation learning algorithms.
Furthermore, we conduct a thorough evaluation of state-of-the-art methods on
the proposed task suite. This evaluation serves as a benchmark for assessing
their capability to learn diverse behaviors. Our findings shed light on the
effectiveness of these methods in tackling the intricate problem of capturing
and generalizing multi-modal human behaviors, offering a valuable reference for
the design of future imitation learning algorithms.
