  Detecting objects from LiDAR point clouds is of tremendous significance in
autonomous driving. In spite of good progress, accurate and reliable 3D
detection is yet to be achieved due to the sparsity and irregularity of LiDAR
point clouds. Among existing strategies, multi-view methods have shown great
promise by leveraging the more comprehensive information from both bird's eye
view (BEV) and range view (RV). These multi-view methods either refine the
proposals predicted from single view via fused features, or fuse the features
without considering the global spatial context; their performance is limited
consequently. In this paper, we propose to adaptively fuse multi-view features
in a global spatial context via Dual Cross-VIew SpaTial Attention (VISTA). The
proposed VISTA is a novel plug-and-play fusion module, wherein the multi-layer
perceptron widely adopted in standard attention modules is replaced with a
convolutional one. Thanks to the learned attention mechanism, VISTA can produce
fused features of high quality for prediction of proposals. We decouple the
classification and regression tasks in VISTA, and an additional constraint of
attention variance is applied that enables the attention module to focus on
specific targets instead of generic points. We conduct thorough experiments on
the benchmarks of nuScenes and Waymo; results confirm the efficacy of our
designs. At the time of submission, our method achieves 63.0% in overall mAP
and 69.8% in NDS on the nuScenes benchmark, outperforming all published methods
by up to 24% in safety-crucial categories such as cyclist. The source code in
PyTorch is available at https://github.com/Gorilla-Lab-SCUT/VISTA
