  One-stage object detectors are trained by optimizing classification-loss and
localization-loss simultaneously, with the former suffering much from extreme
foreground-background class imbalance issue due to the large number of anchors.
This paper alleviates this issue by proposing a novel framework to replace the
classification task in one-stage detectors with a ranking task, and adopting
the Average-Precision loss (AP-loss) for the ranking problem. Due to its
non-differentiability and non-convexity, the AP-loss cannot be optimized
directly. For this purpose, we develop a novel optimization algorithm, which
seamlessly combines the error-driven update scheme in perceptron learning and
backpropagation algorithm in deep networks. We provide in-depth analyses on the
good convergence property and computational complexity of the proposed
algorithm, both theoretically and empirically. Experimental results demonstrate
notable improvement in addressing the imbalance issue in object detection over
existing AP-based optimization algorithms. An improved state-of-the-art
performance is achieved in one-stage detectors based on AP-loss over detectors
using classification-losses on various standard benchmarks. The proposed
framework is also highly versatile in accommodating different network
architectures. Code is available at https://github.com/cccorn/AP-loss .
