  The connection between brain activity and corresponding visual stimuli is
crucial in comprehending the human brain. While deep generative models have
exhibited advancement in recovering brain recordings by generating images
conditioned on fMRI signals, accomplishing high-quality generation with
consistent semantics continues to pose challenges. Moreover, the prediction of
brain activity from visual stimuli remains a formidable undertaking. In this
paper, we introduce a unified framework that addresses both fMRI decoding and
encoding. Commencing with the establishment of two latent spaces capable of
representing and reconstructing fMRI signals and visual images, respectively,
we proceed to align the fMRI signals and visual images within the latent space,
thereby enabling a bidirectional transformation between the two domains. Our
Latent Embedding Alignment (LEA) model concurrently recovers visual stimuli
from fMRI signals and predicts brain activity from images within a unified
framework. The performance of LEA surpasses that of existing methods on
multiple benchmark fMRI decoding and encoding datasets. By integrating fMRI
decoding and encoding, LEA offers a comprehensive solution for modeling the
intricate relationship between brain activity and visual stimuli.
