  The accuracy of recent deep learning based clinical decision support systems
is promising. However, lack of model interpretability remains an obstacle to
widespread adoption of artificial intelligence in healthcare. Using sleep as a
case study, we propose a generalizable method to combine clinical
interpretability with high accuracy derived from black-box deep learning.
Clinician-determined sleep stages from polysomnogram (PSG) remain the gold
standard for evaluating sleep quality. However, PSG manual annotation by
experts is expensive and time-prohibitive. We propose SERF, interpretable Sleep
staging using Embeddings, Rules, and Features to read PSG. SERF provides
interpretation of classified sleep stages through meaningful features derived
from the AASM Manual for the Scoring of Sleep and Associated Events. In SERF,
the embeddings obtained from a hybrid of convolutional and recurrent neural
networks are transposed to the interpretable feature space. These
representative interpretable features are used to train simple models like a
shallow decision tree for classification. Model results are validated on two
publicly available datasets. SERF surpasses the current state-of-the-art for
interpretable sleep staging by 2%. Using Gradient Boosted Trees as the
classifier, SERF obtains 0.766 $\kappa$ and 0.870 AUC-ROC, within 2% of the
current state-of-the-art black-box models.
