  Reconstructing human dynamic vision from brain activity is a challenging task
with great scientific significance. The difficulty stems from two primary
issues: (1) vision-processing mechanisms in the brain are highly intricate and
not fully revealed, making it challenging to directly learn a mapping between
fMRI and video; (2) the temporal resolution of fMRI is significantly lower than
that of natural videos. To overcome these issues, this paper propose a
two-stage model named Mind-Animator, which achieves state-of-the-art
performance on three public datasets. Specifically, during the fMRI-to-feature
stage, we decouple semantic, structural, and motion features from fMRI through
fMRI-vision-language tri-modal contrastive learning and sparse causal
attention. In the feature-to-video stage, these features are merged to videos
by an inflated Stable Diffusion. We substantiate that the reconstructed video
dynamics are indeed derived from fMRI, rather than hallucinations of the
generative model, through permutation tests. Additionally, the visualization of
voxel-wise and ROI-wise importance maps confirms the neurobiological
interpretability of our model.
