  Centred Kernel Alignment (CKA) has recently emerged as a popular metric to
compare activations from biological and artificial neural networks (ANNs) in
order to quantify the alignment between internal representations derived from
stimuli sets (e.g. images, text, video) that are presented to both systems. In
this paper we highlight issues that the community should take into account if
using CKA as an alignment metric with neural data. Neural data are in the
low-data high-dimensionality domain, which is one of the cases where (biased)
CKA results in high similarity scores even for pairs of random matrices. Using
fMRI and MEG data from the THINGS project, we show that if biased CKA is
applied to representations of different sizes in the low-data
high-dimensionality domain, they are not directly comparable due to biased
CKA's sensitivity to differing feature-sample ratios and not stimuli-driven
responses. This situation can arise both when comparing a pre-selected area of
interest (e.g. ROI) to multiple ANN layers, as well as when determining to
which ANN layer multiple regions of interest (ROIs) / sensor groups of
different dimensionality are most similar. We show that biased CKA can be
artificially driven to its maximum value when using independent random data of
different sample-feature ratios. We further show that shuffling sample-feature
pairs of real neural data does not drastically alter biased CKA similarity in
comparison to unshuffled data, indicating an undesirable lack of sensitivity to
stimuli-driven neural responses. Positive alignment of true stimuli-driven
responses is only achieved by using debiased CKA. Lastly, we report findings
that suggest biased CKA is sensitive to the inherent structure of neural data,
only differing from shuffled data when debiased CKA detects stimuli-driven
alignment.
