  Vehicle-to-everything (V2X) communication techniques enable the collaboration
between vehicles and many other entities in the neighboring environment, which
could fundamentally improve the perception system for autonomous driving.
However, the lack of a public dataset significantly restricts the research
progress of collaborative perception. To fill this gap, we present V2X-Sim, a
comprehensive simulated multi-agent perception dataset for V2X-aided autonomous
driving. V2X-Sim provides: (1) \hl{multi-agent} sensor recordings from the
road-side unit (RSU) and multiple vehicles that enable collaborative
perception, (2) multi-modality sensor streams that facilitate multi-modality
perception, and (3) diverse ground truths that support various perception
tasks. Meanwhile, we build an open-source testbed and provide a benchmark for
the state-of-the-art collaborative perception algorithms on three tasks,
including detection, tracking and segmentation. V2X-Sim seeks to stimulate
collaborative perception research for autonomous driving before realistic
datasets become widely available. Our dataset and code are available at
\url{https://ai4ce.github.io/V2X-Sim/}.
