  Decoding EEG during motor imagery is pivotal for the Brain-Computer Interface
(BCI) system, influencing its overall performance significantly. As end-to-end
data-driven learning methods advance, the challenge lies in balancing model
complexity with the need for human interpretability and trust. Despite strides
in EEG-based BCIs, challenges like artefacts and low signal-to-noise ratio
emphasise the ongoing importance of model transparency. This work proposes
using post-hoc explanations to interpret model outcomes and validate them
against domain knowledge. Leveraging the GradCAM post-hoc explanation technique
on the motor imagery dataset, this work demonstrates that relying solely on
accuracy metrics may be inadequate to ensure BCI performance and acceptability.
A model trained using all EEG channels of the dataset achieves 72.60% accuracy,
while a model trained with motor-imagery/movement-relevant channel data has a
statistically insignificant decrease of 1.75%. However, the relevant features
for both are very different based on neurophysiological facts. This work
demonstrates that integrating domain-specific knowledge with XAI techniques
emerges as a promising paradigm for validating the neurophysiological basis of
model outcomes in BCIs. Our results reveal the significance of
neurophysiological validation in evaluating BCI performance, highlighting the
potential risks of exclusively relying on performance metrics when selecting
models for dependable and transparent BCIs.
