  Reliable and accurate 3D object detection is a necessity for safe autonomous
driving. Although LiDAR sensors can provide accurate 3D point cloud estimates
of the environment, they are also prohibitively expensive for many settings.
Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction
in the accuracy gap between methods based on LiDAR sensors and those based on
cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D
depth estimation with those for 3D object detection by converting 2D depth map
outputs to 3D point cloud inputs. However, so far these two networks have to be
trained separately. In this paper, we introduce a new framework based on
differentiable Change of Representation (CoR) modules that allow the entire PL
pipeline to be trained end-to-end. The resulting framework is compatible with
most state-of-the-art networks for both tasks and in combination with PointRCNN
improves over PL consistently across all benchmarks -- yielding the highest
entry on the KITTI image-based 3D object detection leaderboard at the time of
submission. Our code will be made available at
https://github.com/mileyan/pseudo-LiDAR_e2e.
