  A vision-language foundation model pretrained on very large-scale image-text
paired data has the potential to provide generalizable knowledge representation
for downstream visual recognition and detection tasks, especially on
supplementing the undersampled categories in downstream model training. Recent
studies utilizing CLIP for object detection have shown that a two-stage
detector design typically outperforms a one-stage detector, while requiring
more expensive training resources and longer inference time. In this work, we
propose a one-stage detector GridCLIP that narrows its performance gap to those
of two-stage detectors, with approximately 43 and 5 times faster than its
two-stage counterpart (ViLD) in the training and test process respectively.
GridCLIP learns grid-level representations to adapt to the intrinsic principle
of one-stage detection learning by expanding the conventional CLIP image-text
holistic mapping to a more fine-grained, grid-text alignment. This differs from
the region-text mapping in two-stage detectors that apply CLIP directly by
treating regions as images. Specifically, GridCLIP performs Grid-level
Alignment to adapt the CLIP image-level representations to grid-level
representations by aligning to CLIP category representations to learn the
annotated (especially frequent) categories. To learn generalizable visual
representations of broader categories, especially undersampled ones, we perform
Image-level Alignment during training to propagate broad pre-learned categories
in the CLIP image encoder from the image-level to the grid-level
representations. Experiments show that the learned CLIP-based grid-level
representations boost the performance of undersampled (infrequent and novel)
categories, reaching comparable detection performance on the LVIS benchmark.
