  In this paper, we investigate a novel problem of telling the difference
between image pairs in natural language. Compared to previous approaches for
single image captioning, it is challenging to fetch linguistic representation
from two independent visual information. To this end, we have proposed an
effective encoder-decoder caption framework based on Hyper Convolution Net. In
addition, a series of novel feature fusing techniques for pairwise visual
information fusing are introduced and a discriminating referee is proposed to
evaluate the pipeline. Because of the lack of appropriate datasets to support
this task, we have collected and annotated a large new dataset with Amazon
Mechanical Turk (AMT) for generating captions in a pairwise manner (with 14764
images and 26710 image pairs in total). The dataset is the first one on the
relative difference caption task that provides descriptions in free language.
We evaluate the effectiveness of our model on two datasets in the field and it
outperforms the state-of-the-art approach by a large margin.
