  We present a new traffic dataset, METEOR, which captures traffic patterns and
multi-agent driving behaviors in unstructured scenarios. METEOR consists of
more than 1000 one-minute videos, over 2 million annotated frames with bounding
boxes and GPS trajectories for 16 unique agent categories, and more than 13
million bounding boxes for traffic agents. METEOR is a dataset for rare and
interesting, multi-agent driving behaviors that are grouped into traffic
violations, atypical interactions, and diverse scenarios. Every video in METEOR
is tagged using a diverse range of factors corresponding to weather, time of
the day, road conditions, and traffic density. We use METEOR to benchmark
perception methods for object detection and multi-agent behavior prediction.
Our key finding is that state-of-the-art models for object detection and
behavior prediction, which otherwise succeed on existing datasets such as
Waymo, fail on the METEOR dataset. METEOR marks the first step towards the
development of more sophisticated perception models for dense, heterogeneous,
and unstructured scenarios.
