  In this paper, we propose QACE, a new metric based on Question Answering for
Caption Evaluation. QACE generates questions on the evaluated caption and
checks its content by asking the questions on either the reference caption or
the source image. We first develop QACE-Ref that compares the answers of the
evaluated caption to its reference, and report competitive results with the
state-of-the-art metrics. To go further, we propose QACE-Img, which asks the
questions directly on the image, instead of reference. A Visual-QA system is
necessary for QACE-Img. Unfortunately, the standard VQA models are framed as a
classification among only a few thousand categories. Instead, we propose
Visual-T5, an abstractive VQA system. The resulting metric, QACE-Img is
multi-modal, reference-less, and explainable. Our experiments show that
QACE-Img compares favorably w.r.t. other reference-less metrics. We will
release the pre-trained models to compute QACE.
