  The human brain extracts complex information from visual inputs, including
objects, their spatial and semantic interrelations, and their interactions with
the environment. However, a quantitative approach for studying this information
remains elusive. Here, we test whether the contextual information encoded in
large language models (LLMs) is beneficial for modelling the complex visual
information extracted by the brain from natural scenes. We show that LLM
embeddings of scene captions successfully characterise brain activity evoked by
viewing the natural scenes. This mapping captures selectivities of different
brain areas, and is sufficiently robust that accurate scene captions can be
reconstructed from brain activity. Using carefully controlled model
comparisons, we then proceed to show that the accuracy with which LLM
representations match brain representations derives from the ability of LLMs to
integrate complex information contained in scene captions beyond that conveyed
by individual words. Finally, we train deep neural network models to transform
image inputs into LLM representations. Remarkably, these networks learn
representations that are better aligned with brain representations than a large
number of state-of-the-art alternative models, despite being trained on
orders-of-magnitude less data. Overall, our results suggest that LLM embeddings
of scene captions provide a representational format that accounts for complex
information extracted by the brain from visual inputs.
