  The performance of cardiac arrhythmia detection with electrocardiograms(ECGs)
has been considerably improved since the introduction of deep learning models.
In practice, the high performance alone is not sufficient and a proper
explanation is also required. Recently, researchers have started adopting
feature attribution methods to address this requirement, but it has been
unclear which of the methods are appropriate for ECG. In this work, we identify
and customize three evaluation metrics for feature attribution methods based on
the characteristics of ECG: localization score, pointing game, and degradation
score. Using the three evaluation metrics, we evaluate and analyze eleven
widely-used feature attribution methods. We find that some of the feature
attribution methods are much more adequate for explaining ECG, where Grad-CAM
outperforms the second-best method by a large margin.
