  Quantitative modeling of human brain activity based on language
representations has been actively studied in systems neuroscience. However,
previous studies examined word-level representation, and little is known about
whether we could recover structured sentences from brain activity. This study
attempts to generate natural language descriptions of semantic contents from
human brain activity evoked by visual stimuli. To effectively use a small
amount of available brain activity data, our proposed method employs a
pre-trained image-captioning network model using a deep learning framework. To
apply brain activity to the image-captioning network, we train regression
models that learn the relationship between brain activity and deep-layer image
features. The results demonstrate that the proposed model can decode brain
activity and generate descriptions using natural language sentences. We also
conducted several experiments with data from different subsets of brain regions
known to process visual stimuli. The results suggest that semantic information
for sentence generations is widespread across the entire cortex.
