  In this paper, we aim at addressing two critical issues in the 3D detection
task, including the exploitation of multiple sensors~(namely LiDAR point cloud
and camera image), as well as the inconsistency between the localization and
classification confidence. To this end, we propose a novel fusion module to
enhance the point features with semantic image features in a point-wise manner
without any image annotations. Besides, a consistency enforcing loss is
employed to explicitly encourage the consistency of both the localization and
classification confidence. We design an end-to-end learnable framework named
EPNet to integrate these two components. Extensive experiments on the KITTI and
SUN-RGBD datasets demonstrate the superiority of EPNet over the
state-of-the-art methods. Codes and models are available at:
\url{https://github.com/happinesslz/EPNet}.
