  The Image Difference Captioning (IDC) task aims to describe the visual
differences between two similar images with natural language. The major
challenges of this task lie in two aspects: 1) fine-grained visual differences
that require learning stronger vision and language association and 2) high-cost
of manual annotations that leads to limited supervised data. To address these
challenges, we propose a new modeling framework following the
pre-training-finetuning paradigm. Specifically, we design three self-supervised
tasks and contrastive learning strategies to align visual differences and text
descriptions at a fine-grained level. Moreover, we propose a data expansion
strategy to utilize extra cross-task supervision information, such as data for
fine-grained image classification, to alleviate the limitation of available
supervised IDC data. Extensive experiments on two IDC benchmark datasets,
CLEVR-Change and Birds-to-Words, demonstrate the effectiveness of the proposed
modeling framework. The codes and models will be released at
https://github.com/yaolinli/IDC.
