  Brain decoding is a field of computational neuroscience that uses measurable
brain activity to infer mental states or internal representations of perceptual
inputs. Therefore, we propose a novel approach to brain decoding that also
relies on semantic and contextual similarity. We employ an fMRI dataset of
natural image vision and create a deep learning decoding pipeline inspired by
the existence of both bottom-up and top-down processes in human vision. We
train a linear brain-to-feature model to map fMRI activity features to visual
stimuli features, assuming that the brain projects visual information onto a
space that is homeomorphic to the latent space represented by the last
convolutional layer of a pretrained convolutional neural network, which
typically collects a variety of semantic features that summarize and highlight
similarities and differences between concepts. These features are then
categorized in the latent space using a nearest-neighbor strategy, and the
results are used to condition a generative latent diffusion model to create
novel images. From fMRI data only, we produce reconstructions of visual stimuli
that match the original content very well on a semantic level, surpassing the
state of the art in previous literature. We evaluate our work and obtain good
results using a quantitative semantic metric (the Wu-Palmer similarity metric
over the WordNet lexicon, which had an average value of 0.57) and perform a
human evaluation experiment that resulted in correct evaluation, according to
the multiplicity of human criteria in evaluating image similarity, in over 80%
of the test set.
