  In this paper, we focus on exploring the fusion of images and point clouds
for 3D object detection in view of the complementary nature of the two
modalities, i.e., images possess more semantic information while point clouds
specialize in distance sensing. To this end, we present a novel two-stage
multi-modal fusion network for 3D object detection, taking both binocular
images and raw point clouds as input. The whole architecture facilitates
two-stage fusion. The first stage aims at producing 3D proposals through sparse
point-wise feature fusion. Within the first stage, we further exploit a joint
anchor mechanism that enables the network to utilize 2D-3D classification and
regression simultaneously for better proposal generation.
  The second stage works on the 2D and 3D proposal regions and fuses their
dense features. In addition, we propose to use pseudo LiDAR points from stereo
matching as a data augmentation method to densify the LiDAR points, as we
observe that objects missed by the detection network mostly have too few points
especially for far-away objects. Our experiments on the KITTI dataset show that
the proposed multi-stage fusion helps the network to learn better
representations.
