  Concurrent perception datasets for autonomous driving are mainly limited to
frontal view with sensors mounted on the vehicle. None of them is designed for
the overlooked roadside perception tasks. On the other hand, the data captured
from roadside cameras have strengths over frontal-view data, which is believed
to facilitate a safer and more intelligent autonomous driving system. To
accelerate the progress of roadside perception, we present the first
high-diversity challenging Roadside Perception 3D dataset- Rope3D from a novel
view. The dataset consists of 50k images and over 1.5M 3D objects in various
scenes, which are captured under different settings including various cameras
with ambiguous mounting positions, camera specifications, viewpoints, and
different environmental conditions. We conduct strict 2D-3D joint annotation
and comprehensive data analysis, as well as set up a new 3D roadside perception
benchmark with metrics and evaluation devkit. Furthermore, we tailor the
existing frontal-view monocular 3D object detection approaches and propose to
leverage the geometry constraint to solve the inherent ambiguities caused by
various sensors, viewpoints. Our dataset is available on
https://thudair.baai.ac.cn/rope.
