  Modern detection transformers (DETRs) use a set of object queries to predict
a list of bounding boxes, sort them by their classification confidence scores,
and select the top-ranked predictions as the final detection results for the
given input image. A highly performant object detector requires accurate
ranking for the bounding box predictions. For DETR-based detectors, the
top-ranked bounding boxes suffer from less accurate localization quality due to
the misalignment between classification scores and localization accuracy, thus
impeding the construction of high-quality detectors. In this work, we introduce
a simple and highly performant DETR-based object detector by proposing a series
of rank-oriented designs, combinedly called Rank-DETR. Our key contributions
include: (i) a rank-oriented architecture design that can prompt positive
predictions and suppress the negative ones to ensure lower false positive
rates, as well as (ii) a rank-oriented loss function and matching cost design
that prioritizes predictions of more accurate localization accuracy during
ranking to boost the AP under high IoU thresholds. We apply our method to
improve the recent SOTA methods (e.g., H-DETR and DINO-DETR) and report strong
COCO object detection results when using different backbones such as
ResNet-$50$, Swin-T, and Swin-L, demonstrating the effectiveness of our
approach. Code is available at \url{https://github.com/LeapLabTHU/Rank-DETR}.
