  Camera-based 3D object detection in BEV (Bird's Eye View) space has drawn
great attention over the past few years. Dense detectors typically follow a
two-stage pipeline by first constructing a dense BEV feature and then
performing object detection in BEV space, which suffers from complex view
transformations and high computation cost. On the other side, sparse detectors
follow a query-based paradigm without explicit dense BEV feature construction,
but achieve worse performance than the dense counterparts. In this paper, we
find that the key to mitigate this performance gap is the adaptability of the
detector in both BEV and image space. To achieve this goal, we propose
SparseBEV, a fully sparse 3D object detector that outperforms the dense
counterparts. SparseBEV contains three key designs, which are (1)
scale-adaptive self attention to aggregate features with adaptive receptive
field in BEV space, (2) adaptive spatio-temporal sampling to generate sampling
locations under the guidance of queries, and (3) adaptive mixing to decode the
sampled features with dynamic weights from the queries. On the test split of
nuScenes, SparseBEV achieves the state-of-the-art performance of 67.5 NDS. On
the val split, SparseBEV achieves 55.8 NDS while maintaining a real-time
inference speed of 23.5 FPS. Code is available at
https://github.com/MCG-NJU/SparseBEV.
