  Image2Speech is the relatively new task of generating a spoken description of
an image. This paper presents an investigation into the evaluation of this
task. For this, first an Image2Speech system was implemented which generates
image captions consisting of phoneme sequences. This system outperformed the
original Image2Speech system on the Flickr8k corpus. Subsequently, these
phoneme captions were converted into sentences of words. The captions were
rated by human evaluators for their goodness of describing the image. Finally,
several objective metric scores of the results were correlated with these human
ratings. Although BLEU4 does not perfectly correlate with human ratings, it
obtained the highest correlation among the investigated metrics, and is the
best currently existing metric for the Image2Speech task. Current metrics are
limited by the fact that they assume their input to be words. A more
appropriate metric for the Image2Speech task should assume its input to be
parts of words, i.e. phonemes, instead.
