  Bounding box regression is an important component in object detection. Recent
work achieves promising performance by optimizing the Intersection over
Union~(IoU). However, IoU-based loss has the gradient vanish problem in the
case of low overlapping bounding boxes, and the model could easily ignore these
simple cases. In this paper, we propose Side Overlap~(SO) loss by maximizing
the side overlap of two bounding boxes, which puts more penalty for low
overlapping bounding box cases. Besides, to speed up the convergence, the
Corner Distance~(CD) is added into the objective function. Combining the Side
Overlap and Corner Distance, we get a new regression objective function,
\textit{Side and Corner Align Loss~(SCALoss)}. The SCALoss is well-correlated
with IoU loss, which also benefits the evaluation metric but produces more
penalty for low-overlapping cases. It can serve as a comprehensive similarity
measure, leading to better localization performance and faster convergence
speed. Experiments on COCO, PASCAL VOC, and LVIS benchmarks show that SCALoss
can bring consistent improvement and outperform $\ell_n$ loss and IoU based
loss with popular object detectors such as YOLOV3, SSD, Faster-RCNN. Code is
available at: \url{https://github.com/Turoad/SCALoss}.
