  The CLIP model has been recently proven to be very effective for a variety of
cross-modal tasks, including the evaluation of captions generated from
vision-and-language architectures. In this paper, we propose a new recipe for a
contrastive-based evaluation metric for image captioning, namely
Positive-Augmented Contrastive learning Score (PAC-S), that in a novel way
unifies the learning of a contrastive visual-semantic space with the addition
of generated images and text on curated data. Experiments spanning several
datasets demonstrate that our new metric achieves the highest correlation with
human judgments on both images and videos, outperforming existing
reference-based metrics like CIDEr and SPICE and reference-free metrics like
CLIP-Score. Finally, we test the system-level correlation of the proposed
metric when considering popular image captioning approaches, and assess the
impact of employing different cross-modal features. Our source code and trained
models are publicly available at: https://github.com/aimagelab/pacscore.
