  Within neuroimgaing studies it is a common practice to perform repetitions of
trials in an experiment when working with a noisy class of data acquisition
system, such as electroencephalography (EEG) or magnetoencephalography (MEG).
While this approach can be useful in some experimental designs, it presents
significant limitations for certain types of analyses, such as identifying the
category of an object observed by a subject. In this study we demonstrate that
when trials relating to a single object are allowed to appear in both the
training and testing sets, almost any classification algorithm is capable of
learning the representation of an object given only category labels. This
ability to learn object representations is of particular significance as it
suggests that the results of several published studies which predict the
category of observed objects from EEG signals may be affected by a subtle form
of leakage which has inflated their reported accuracies. We demonstrate the
ability of both simple classification algorithms, and sophisticated deep
learning models, to learn object representations given only category labels. We
do this using two datasets; the Kaneshiro et al. (2015) dataset and the Gifford
et al. (2022) dataset. Our results raise doubts about the true generalizability
of several published models and suggests that the reported performance of these
models may be significantly inflated.
