  News Image Captioning requires describing an image by leveraging additional
context from a news article. Previous works only coarsely leverage the article
to extract the necessary context, which makes it challenging for models to
identify relevant events and named entities. In our paper, we first demonstrate
that by combining more fine-grained context that captures the key named
entities (obtained via an oracle) and the global context that summarizes the
news, we can dramatically improve the model's ability to generate accurate news
captions. This begs the question, how to automatically extract such key
entities from an image? We propose to use the pre-trained vision and language
retrieval model CLIP to localize the visually grounded entities in the news
article and then capture the non-visual entities via an open relation
extraction model. Our experiments demonstrate that by simply selecting a better
context from the article, we can significantly improve the performance of
existing models and achieve new state-of-the-art performance on multiple
benchmarks.
