  The training paradigm of DETRs is heavily contingent upon pre-training their
backbone on the ImageNet dataset. However, the limited supervisory signals
provided by the image classification task and one-to-one matching strategy
result in an inadequately pre-trained neck for DETRs. Additionally, the
instability of matching in the early stages of training engenders
inconsistencies in the optimization objectives of DETRs. To address these
issues, we have devised an innovative training methodology termed step-by-step
training. Specifically, in the first stage of training, we employ a classic
detector, pre-trained with a one-to-many matching strategy, to initialize the
backbone and neck of the end-to-end detector. In the second stage of training,
we froze the backbone and neck of the end-to-end detector, necessitating the
training of the decoder from scratch. Through the application of step-by-step
training, we have introduced the first real-time end-to-end object detection
model that utilizes a purely convolutional structure encoder, DETR with YOLO
(DEYO). Without reliance on any supplementary training data, DEYO surpasses all
existing real-time object detectors in both speed and accuracy. Moreover, the
comprehensive DEYO series can complete its second-phase training on the COCO
dataset using a single 8GB RTX 4060 GPU, significantly reducing the training
expenditure. Source code and pre-trained models are available at
https://github.com/ouyanghaodong/DEYO.
