  Object detection has recently achieved a breakthrough for removing the last
one non-differentiable component in the pipeline, Non-Maximum Suppression
(NMS), and building up an end-to-end system. However, what makes for its
one-to-one prediction has not been well understood. In this paper, we first
point out that one-to-one positive sample assignment is the key factor, while,
one-to-many assignment in previous detectors causes redundant predictions in
inference. Second, we surprisingly find that even training with one-to-one
assignment, previous detectors still produce redundant predictions. We identify
that classification cost in matching cost is the main ingredient: (1) previous
detectors only consider location cost, (2) by additionally introducing
classification cost, previous detectors immediately produce one-to-one
prediction during inference. We introduce the concept of score gap to explore
the effect of matching cost. Classification cost enlarges the score gap by
choosing positive samples as those of highest score in the training iteration
and reducing noisy positive samples brought by only location cost. Finally, we
demonstrate the advantages of end-to-end object detection on crowded scenes.
The code is available at: \url{https://github.com/PeizeSun/OneNet}.
