  Understanding how the brain encodes external stimuli and how these stimuli
can be decoded from the measured brain activities are long-standing and
challenging questions in neuroscience. In this paper, we focus on
reconstructing the complex image stimuli from fMRI (functional magnetic
resonance imaging) signals. Unlike previous works that reconstruct images with
single objects or simple shapes, our work aims to reconstruct image stimuli
that are rich in semantics, closer to everyday scenes, and can reveal more
perspectives. However, data scarcity of fMRI datasets is the main obstacle to
applying state-of-the-art deep learning models to this problem. We find that
incorporating an additional text modality is beneficial for the reconstruction
problem compared to directly translating brain signals to images. Therefore,
the modalities involved in our method are: (i) voxel-level fMRI signals, (ii)
observed images that trigger the brain signals, and (iii) textual description
of the images. To further address data scarcity, we leverage an aligned
vision-language latent space pre-trained on massive datasets. Instead of
training models from scratch to find a latent space shared by the three
modalities, we encode fMRI signals into this pre-aligned latent space. Then,
conditioned on embeddings in this space, we reconstruct images with a
generative model. The reconstructed images from our pipeline balance both
naturalness and fidelity: they are photo-realistic and capture the ground truth
image contents well.
