  Self-supervised pre-training and transformer-based networks have
significantly improved the performance of object detection. However, most of
the current self-supervised object detection methods are built on
convolutional-based architectures. We believe that the transformers' sequence
characteristics should be considered when designing a transformer-based
self-supervised method for the object detection task. To this end, we propose
SeqCo-DETR, a novel Sequence Consistency-based self-supervised method for
object DEtection with TRansformers. SeqCo-DETR defines a simple but effective
pretext by minimizes the discrepancy of the output sequences of transformers
with different image views as input and leverages bipartite matching to find
the most relevant sequence pairs to improve the sequence-level self-supervised
representation learning performance. Furthermore, we provide a mask-based
augmentation strategy incorporated with the sequence consistency strategy to
extract more representative contextual information about the object for the
object detection task. Our method achieves state-of-the-art results on MS COCO
(45.8 AP) and PASCAL VOC (64.1 AP), demonstrating the effectiveness of our
approach.
