  Deep learning-based support systems have demonstrated encouraging results in
numerous clinical applications involving the processing of time series data.
While such systems often are very accurate, they have no inherent mechanism for
explaining what influenced the predictions, which is critical for clinical
tasks. However, existing explainability techniques lack an important component
for trustworthy and reliable decision support, namely a notion of uncertainty.
In this paper, we address this lack of uncertainty by proposing a deep ensemble
approach where a collection of DNNs are trained independently. A measure of
uncertainty in the relevance scores is computed by taking the standard
deviation across the relevance scores produced by each model in the ensemble,
which in turn is used to make the explanations more reliable. The class
activation mapping method is used to assign a relevance score for each time
step in the time series. Results demonstrate that the proposed ensemble is more
accurate in locating relevant time steps and is more consistent across random
initializations, thus making the model more trustworthy. The proposed
methodology paves the way for constructing trustworthy and dependable support
systems for processing clinical time series for healthcare related tasks.
