  Modern 3D object detectors have immensely benefited from the end-to-end
learning idea. However, most of them use a post-processing algorithm called
Non-Maximal Suppression (NMS) only during inference. While there were attempts
to include NMS in the training pipeline for tasks such as 2D object detection,
they have been less widely adopted due to a non-mathematical expression of the
NMS. In this paper, we present and integrate GrooMeD-NMS -- a novel Grouped
Mathematically Differentiable NMS for monocular 3D object detection, such that
the network is trained end-to-end with a loss on the boxes after NMS. We first
formulate NMS as a matrix operation and then group and mask the boxes in an
unsupervised manner to obtain a simple closed-form expression of the NMS.
GrooMeD-NMS addresses the mismatch between training and inference pipelines
and, therefore, forces the network to select the best 3D box in a
differentiable manner. As a result, GrooMeD-NMS achieves state-of-the-art
monocular 3D object detection results on the KITTI benchmark dataset performing
comparably to monocular video-based methods. Code and models at
https://github.com/abhi1kumar/groomed_nms
