  Vision models with high overall accuracy often exhibit systematic errors in
specific scenarios, posing potential serious safety concerns. Diagnosing bugs
of vision models is gaining increased attention, however traditional diagnostic
approaches require annotation efforts (eg rich metadata accompanying each
samples of CelebA). To address this issue,We propose a language-assisted
diagnostic method that uses texts instead of images to diagnose bugs in vision
models based on multi-modal models (eg CLIP). Our approach connects the
embedding space of CLIP with the buggy vision model to be diagnosed; meanwhile,
utilizing a shared classifier and the cross-modal transferability of embedding
space from CLIP, the text-branch of CLIP become a proxy model to find bugs in
the buggy model. The proxy model can classify texts paired with images. During
the diagnosis, a Large Language Model (LLM) is employed to obtain task-relevant
corpora, and this corpora is used to extract keywords. Descriptions constructed
with templates containing these keywords serve as input text to probe errors in
the proxy model. Finally, we validate the ability to diagnose existing visual
models using language on the Waterbirds and CelebA datasets, we can identify
bugs comprehensible to human experts, uncovering not only known bugs but also
previously unknown ones.
