  Large-scale datasets have fueled recent advancements in AI-based autonomous
vehicle research. However, these datasets are usually collected from a single
vehicle's one-time pass of a certain location, lacking multiagent interactions
or repeated traversals of the same place. Such information could lead to
transformative enhancements in autonomous vehicles' perception, prediction, and
planning capabilities. To bridge this gap, in collaboration with the
self-driving company May Mobility, we present the MARS dataset which unifies
scenarios that enable MultiAgent, multitraveRSal, and multimodal autonomous
vehicle research. More specifically, MARS is collected with a fleet of
autonomous vehicles driving within a certain geographical area. Each vehicle
has its own route and different vehicles may appear at nearby locations. Each
vehicle is equipped with a LiDAR and surround-view RGB cameras. We curate two
subsets in MARS: one facilitates collaborative driving with multiple vehicles
simultaneously present at the same location, and the other enables memory
retrospection through asynchronous traversals of the same location by multiple
vehicles. We conduct experiments in place recognition and neural
reconstruction. More importantly, MARS introduces new research opportunities
and challenges such as multitraversal 3D reconstruction, multiagent perception,
and unsupervised object discovery. Our data and codes can be found at
https://ai4ce.github.io/MARS/.
