  In recent years, deep learning has shown potential and efficiency in a wide
area including computer vision, image and signal processing. Yet, translational
challenges remain for user applications due to a lack of interpretability of
algorithmic decisions and results. This black box problem is particularly
problematic for high-risk applications such as medical-related decision-making.
The current study goal was to design an interpretable deep learning system for
time series classification of electroencephalogram (EEG) for sleep stage
scoring as a step toward designing a transparent system. We have developed an
interpretable deep neural network that includes a kernel-based layer based on a
set of principles used for sleep scoring by human experts in the visual
analysis of polysomnographic records. A kernel-based convolutional layer was
defined and used as the first layer of the system and made available for user
interpretation. The trained system and its results were interpreted in four
levels from the microstructure of EEG signals, such as trained kernels and the
effect of each kernel on the detected stages, to macrostructures, such as the
transition between stages. The proposed system demonstrated greater performance
than prior studies and the results of interpretation showed that the system
learned information which was consistent with expert knowledge.
