  Pretrained cross-modal models, for instance, the most representative CLIP,
have recently led to a boom in using pre-trained models for cross-modal
zero-shot tasks, considering the generalization properties. However, we
analytically discover that CLIP suffers from the text-to-image retrieval
hallucination, adversely limiting its capabilities under zero-shot learning:
CLIP would select the image with the highest score when asked to figure out
which image perfectly matches one given query text among several candidate
images even though CLIP knows contents in the image. Accordingly, we propose a
Balanced Score with Auxiliary Prompts (BSAP) to mitigate the CLIP's
text-to-image retrieval hallucination under zero-shot learning. Specifically,
we first design auxiliary prompts to provide multiple reference outcomes for
every single image retrieval, then the outcomes derived from each retrieved
image in conjunction with the target text are normalized to obtain the final
similarity, which alleviates hallucinations in the model. Additionally, we can
merge CLIP's original results and BSAP to obtain a more robust hybrid outcome
(BSAP-H). Extensive experiments on two typical zero-shot learning tasks, i.e.,
Referring Expression Comprehension (REC) and Referring Image Segmentation
(RIS), are conducted to demonstrate the effectiveness of our BSAP.
Specifically, when evaluated on the validation dataset of RefCOCO in REC, BSAP
increases CLIP's performance by 20.6%. Further, we validate that our strategy
could be applied in other types of pretrained cross-modal models, such as ALBEF
and BLIP.
