  Large language models (LLMs) and large multimodal models (LMMs) have
significantly impacted the AI community, industry, and various economic
sectors. In journalism, integrating AI poses unique challenges and
opportunities, particularly in enhancing the quality and efficiency of news
reporting. This study explores how LLMs and LMMs can assist journalistic
practice by generating contextualised captions for images accompanying news
articles. We conducted experiments using the GoodNews dataset to evaluate the
ability of LMMs (BLIP-2, GPT-4v, or LLaVA) to incorporate one of two types of
context: entire news articles, or extracted named entities. In addition, we
compared their performance to a two-stage pipeline composed of a captioning
model (BLIP-2, OFA, or ViT-GPT2) with post-hoc contextualisation with LLMs
(GPT-4 or LLaMA). We assess a diversity of models, and we find that while the
choice of contextualisation model is a significant factor for the two-stage
pipelines, this is not the case in the LMMs, where smaller, open-source models
perform well compared to proprietary, GPT-powered ones. Additionally, we found
that controlling the amount of provided context enhances performance. These
results highlight the limitations of a fully automated approach and underscore
the necessity for an interactive, human-in-the-loop strategy.
