  In this paper, we present an Intersection-over-Union (IoU) guided two-stage
3D object detector with a voxel-to-point decoder. To preserve the necessary
information from all raw points and maintain the high box recall in voxel based
Region Proposal Network (RPN), we propose a residual voxel-to-point decoder to
extract the point features in addition to the map-view features from the voxel
based RPN. We use a 3D Region of Interest (RoI) alignment to crop and align the
features with the proposal boxes for accurately perceiving the object position.
The RoI-Aligned features are finally aggregated with the corner geometry
embeddings that can provide the potentially missing corner information in the
box refinement stage. We propose a simple and efficient method to align the
estimated IoUs to the refined proposal boxes as a more relevant localization
confidence. The comprehensive experiments on KITTI and Waymo Open Dataset
demonstrate that our method achieves significant improvements with novel
architectures against the existing methods. The code is available on Github
URL\footnote{\url{https://github.com/jialeli1/From-Voxel-to-Point}}.
