  Recent advancements in Vehicle-to-Everything communication technology have
enabled autonomous vehicles to share sensory information to obtain better
perception performance. With the rapid growth of autonomous vehicles and
intelligent infrastructure, the V2X perception systems will soon be deployed at
scale, which raises a safety-critical question: \textit{how can we evaluate and
improve its performance under challenging traffic scenarios before the
real-world deployment?} Collecting diverse large-scale real-world test scenes
seems to be the most straightforward solution, but it is expensive and
time-consuming, and the collections can only cover limited scenarios. To this
end, we propose the first open adversarial scene generator V2XP-ASG that can
produce realistic, challenging scenes for modern LiDAR-based multi-agent
perception systems. V2XP-ASG learns to construct an adversarial collaboration
graph and simultaneously perturb multiple agents' poses in an adversarial and
plausible manner. The experiments demonstrate that V2XP-ASG can effectively
identify challenging scenes for a large range of V2X perception systems.
Meanwhile, by training on the limited number of generated challenging scenes,
the accuracy of V2X perception systems can be further improved by 12.3\% on
challenging and 4\% on normal scenes. Our code will be released at
https://github.com/XHwind/V2XP-ASG.
