  Decoding and reconstructing images from brain imaging data is a research area
of high interest. Recent progress in deep generative neural networks has
introduced new opportunities to tackle this problem. Here, we employ a recently
proposed large-scale bi-directional generative adversarial network, called
BigBiGAN, to decode and reconstruct natural scenes from fMRI patterns. BigBiGAN
converts images into a 120-dimensional latent space which encodes class and
attribute information together, and can also reconstruct images based on their
latent vectors. We computed a linear mapping between fMRI data, acquired over
images from 150 different categories of ImageNet, and their corresponding
BigBiGAN latent vectors. Then, we applied this mapping to the fMRI activity
patterns obtained from 50 new test images from 50 unseen categories in order to
retrieve their latent vectors, and reconstruct the corresponding images.
Pairwise image decoding from the predicted latent vectors was highly accurate
(84%). Moreover, qualitative and quantitative assessments revealed that the
resulting image reconstructions were visually plausible, successfully captured
many attributes of the original images, and had high perceptual similarity with
the original content. This method establishes a new state-of-the-art for
fMRI-based natural image reconstruction, and can be flexibly updated to take
into account any future improvements in generative models of natural scene
images.
