  Image captioning models require the high-level generalization ability to
describe the contents of various images in words. Most existing approaches
treat the image-caption pairs equally in their training without considering the
differences in their learning difficulties. Several image captioning approaches
introduce curriculum learning methods that present training data with
increasing levels of difficulty. However, their difficulty measurements are
either based on domain-specific features or prior model training. In this
paper, we propose a simple yet efficient difficulty measurement for image
captioning using cross-modal similarity calculated by a pretrained
vision-language model. Experiments on the COCO and Flickr30k datasets show that
our proposed approach achieves superior performance and competitive convergence
speed to baselines without requiring heuristics or incurring additional
training costs. Moreover, the higher model performance on difficult examples
and unseen data also demonstrates the generalization ability.
