  Decoding images from non-invasive electroencephalographic (EEG) signals has
been a grand challenge in understanding how the human brain process visual
information in real-world scenarios. To cope with the issues of signal-to-noise
ratio and nonstationarity, this paper introduces a MUltimodal
Similarity-keeping contrastivE learning (MUSE) framework for zero-shot
EEG-based image classification. We develop a series of multivariate time-series
encoders tailored for EEG signals and assess the efficacy of regularized
contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our
method achieves state-of-the-art performance, with a top-1 accuracy of 19.3%
and a top-5 accuracy of 48.8% in 200-way zero-shot image classification.
Furthermore, we visualize neural patterns via model interpretation, shedding
light on the visual processing dynamics in the human brain. The code repository
for this work is available at: https://github.com/ChiShengChen/MUSE_EEG.
