  Image Captioning is a current research task to describe the image content
using the objects and their relationships in the scene. To tackle this task,
two important research areas converge, artificial vision, and natural language
processing. In Image Captioning, as in any computational intelligence task, the
performance metrics are crucial for knowing how well (or bad) a method
performs. In recent years, it has been observed that classical metrics based on
n-grams are insufficient to capture the semantics and the critical meaning to
describe the content in an image. Looking to measure how well or not the set of
current and more recent metrics are doing, in this article, we present an
evaluation of several kinds of Image Captioning metrics and a comparison
between them using the well-known MS COCO dataset. The metrics were selected
from the most used in prior works, they are those based on $n$-grams as BLEU,
SacreBLEU, METEOR, ROGUE-L, CIDEr, SPICE, and those based on embeddings, such
as BERTScore and CLIPScore. For this, we designed two scenarios; 1) a set of
artificially build captions with several qualities, and 2) a comparison of some
state-of-the-art Image Captioning methods. Interesting findings were found
trying to answer the questions: Are the current metrics helping to produce
high-quality captions? How do actual metrics compare to each other? What are
the metrics really measuring?
