  Existing approaches to modeling associations between visual stimuli and brain
responses are facing difficulties in handling between-subject variance and
model generalization. Inspired by the recent progress in modeling speech-brain
response, we propose in this work a "match-vs-mismatch" deep learning model to
classify whether a video clip induces excitatory responses in recorded EEG
signals and learn associations between the visual content and corresponding
neural recordings. Using an exclusive experimental dataset, we demonstrate that
the proposed model is able to achieve the highest accuracy on unseen subjects
as compared to other baseline models. Furthermore, we analyze the inter-subject
noise using a subject-level silhouette score in the embedding space and show
that the developed model is able to mitigate inter-subject noise and
significantly reduce the silhouette score. Moreover, we examine the Grad-CAM
activation score and show that the brain regions associated with language
processing contribute most to the model predictions, followed by regions
associated with visual processing. These results have the potential to
facilitate the development of neural recording-based video reconstruction and
its related applications.
