  Training a referring expression comprehension (ReC) model for a new visual
domain requires collecting referring expressions, and potentially corresponding
bounding boxes, for images in the domain. While large-scale pre-trained models
are useful for image classification across domains, it remains unclear if they
can be applied in a zero-shot manner to more complex tasks like ReC. We present
ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a
state-of-the-art large-scale model, for ReC. Motivated by the close connection
between ReC and CLIP's contrastive pre-training objective, the first component
of ReCLIP is a region-scoring method that isolates object proposals via
cropping and blurring, and passes them to CLIP. However, through controlled
experiments on a synthetic dataset, we find that CLIP is largely incapable of
performing spatial reasoning off-the-shelf. Thus, the second component of
ReCLIP is a spatial relation resolver that handles several types of spatial
relations. We reduce the gap between zero-shot baselines from prior work and
supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game
imagery), ReCLIP's relative improvement over supervised ReC models trained on
real images is 8%.
