  We present a conceptually simple, efficient, and general framework for
localization problems in DETR-like models. We add plugins to well-trained
models instead of inefficiently designing new models and training them from
scratch. The method, called RefineBox, refines the outputs of DETR-like
detectors by lightweight refinement networks. RefineBox is easy to implement
and train as it only leverages the features and predicted boxes from the
well-trained detection models. Our method is also efficient as we freeze the
trained detectors during training. In addition, we can easily generalize
RefineBox to various trained detection models without any modification. We
conduct experiments on COCO and LVIS $1.0$. Experimental results indicate the
effectiveness of our RefineBox for DETR and its representative variants (Figure
1). For example, the performance gains for DETR, Conditinal-DETR, DAB-DETR, and
DN-DETR are 2.4 AP, 2.5 AP, 1.9 AP, and 1.6 AP, respectively. We hope our work
will bring the attention of the detection community to the localization
bottleneck of current DETR-like models and highlight the potential of the
RefineBox framework. Code and models will be publicly available at:
\href{https://github.com/YiqunChen1999/RefineBox}{https://github.com/YiqunChen1999/RefineBox}.
