  Understanding the hidden mechanisms behind human's visual perception is a
fundamental question in neuroscience. To that end, investigating into the
neural responses of human mind activities, such as functional Magnetic
Resonance Imaging (fMRI), has been a significant research vehicle. However,
analyzing fMRI signals is challenging, costly, daunting, and demanding for
professional training. Despite remarkable progress in artificial intelligence
(AI) based fMRI analysis, existing solutions are limited and far away from
being biologically meaningful and practically useful. In this context, we leap
forward to demonstrate how AI can go beyond the current state of the art by
advancing from 2D visuals to visually plausible and functionally more
comprehensive 3D visuals decoded from brain signals, enabling automatic more
sophisticated modeling of fMRI data. Innovationally, we reformulate the task of
analyzing fMRI data as a conditional 3D object generation problem. We design a
novel 3D object representation learning method, Brain3D, that takes as input
the fMRI data of a subject who was presented with a 2D image, and yields as
output the corresponding 3D object visuals. Importantly, we show that our AI
agent captures the distinct functionalities of each region of human vision
system as well as their intricate interplay relationships, aligning remarkably
with the established discoveries of neuroscience. Non-expert diagnosis indicate
that \ourmodel{} can successfully identify the disordered brain regions in
simulated scenarios, such as V1, V2, V3, V4, and the medial temporal lobe (MTL)
within the human visual system. We also present results in cross-modal 3D
visual generation setting, showcasing the perception quality of our 3D
generation.
