  Although CLIPScore is a powerful generic metric that captures the similarity
between a text and an image, it fails to distinguish between a caption that is
meant to complement the information in an image and a description that is meant
to replace an image entirely, e.g., for accessibility. We address this
shortcoming by updating the CLIP model with the Concadia dataset to assign
higher scores to descriptions than captions using parameter efficient
fine-tuning and a loss objective derived from work on causal interpretability.
This model correlates with the judgements of blind and low-vision people while
preserving transfer capabilities and has interpretable structure that sheds
light on the caption--description distinction.
