  Research efforts to understand neural signals have been ongoing for many
years, with visual decoding from fMRI signals attracting considerable
attention. Particularly, the advent of image diffusion models has advanced the
reconstruction of images from fMRI data significantly. However, existing
approaches often introduce inter- and intra- subject variations in the
reconstructed images, which can compromise accuracy. To address current
limitations in multi-subject brain decoding, we introduce a new Transformer
architecture called MindFormer. This model is specifically designed to generate
fMRI-conditioned feature vectors that can be used for conditioning Stable
Diffusion model. More specifically, MindFormer incorporates two key
innovations: 1) a novel training strategy based on the IP-Adapter to extract
semantically meaningful features from fMRI signals, and 2) a subject specific
token and linear layer that effectively capture individual differences in fMRI
signals while synergistically combines multi subject fMRI data for training.
Our experimental results demonstrate that Stable Diffusion, when integrated
with MindFormer, produces semantically consistent images across different
subjects. This capability significantly surpasses existing models in
multi-subject brain decoding. Such advancements not only improve the accuracy
of our reconstructions but also deepen our understanding of neural processing
variations among individuals.
