  Drawing inspiration from the hierarchical processing of the human auditory
system, which transforms sound from low-level acoustic features to high-level
semantic understanding, we introduce a novel coarse-to-fine audio
reconstruction method. Leveraging non-invasive functional Magnetic Resonance
Imaging (fMRI) data, our approach mimics the inverse pathway of auditory
processing. Initially, we utilize CLAP to decode fMRI data coarsely into a
low-dimensional semantic space, followed by a fine-grained decoding into the
high-dimensional AudioMAE latent space guided by semantic features. These
fine-grained neural features serve as conditions for audio reconstruction
through a Latent Diffusion Model (LDM). Validation on three public fMRI
datasets-Brain2Sound, Brain2Music, and Brain2Speech-underscores the superiority
of our coarse-to-fine decoding method over stand-alone fine-grained approaches,
showcasing state-of-the-art performance in metrics like FD, FAD, and KL.
Moreover, by employing semantic prompts during decoding, we enhance the quality
of reconstructed audio when semantic features are suboptimal. The demonstrated
versatility of our model across diverse stimuli highlights its potential as a
universal brain-to-audio framework. This research contributes to the
comprehension of the human auditory system, pushing boundaries in neural
decoding and audio reconstruction methodologies.
