  Recent major milestones have successfully decoded non-invasive brain signals
(e.g. functional Magnetic Resonance Imaging (fMRI) and electroencephalogram
(EEG)) into natural language. Despite the progress in model design, how to
split the datasets for training, validating, and testing still remains a matter
of debate. Most of the prior researches applied subject-specific data
splitting, where the decoding model is trained and evaluated per subject. Such
splitting method poses challenges to the utilization efficiency of dataset as
well as the generalization of models. In this study, we propose a cross-subject
data splitting criterion for brain-to-text decoding on various types of
cognitive dataset (fMRI, EEG), aiming to maximize dataset utilization and
improve model generalization. We undertake a comprehensive analysis on existing
cross-subject data splitting strategies and prove that all these methods suffer
from data leakage, namely the leakage of test data to training set, which
significantly leads to overfitting and overestimation of decoding models. The
proposed cross-subject splitting method successfully addresses the data leakage
problem and we re-evaluate some SOTA brain-to-text decoding models as baselines
for further research.
