  Unpacking and comprehending how black-box machine learning algorithms make
decisions has been a persistent challenge for researchers and end-users.
Explaining time-series predictive models is useful for clinical applications
with high stakes to understand the behavior of prediction models. However,
existing approaches to explain such models are frequently unique to data where
the features do not have a time-varying component. In this paper, we introduce
WindowSHAP, a model-agnostic framework for explaining time-series classifiers
using Shapley values. We intend for WindowSHAP to mitigate the computational
complexity of calculating Shapley values for long time-series data as well as
improve the quality of explanations. WindowSHAP is based on partitioning a
sequence into time windows. Under this framework, we present three distinct
algorithms of Stationary, Sliding and Dynamic WindowSHAP, each evaluated
against baseline approaches, KernelSHAP and TimeSHAP, using perturbation and
sequence analyses metrics. We applied our framework to clinical time-series
data from both a specialized clinical domain (Traumatic Brain Injury - TBI) as
well as a broad clinical domain (critical care medicine). The experimental
results demonstrate that, based on the two quantitative metrics, our framework
is superior at explaining clinical time-series classifiers, while also reducing
the complexity of computations. We show that for time-series data with 120 time
steps (hours), merging 10 adjacent time points can reduce the CPU time of
WindowSHAP by 80% compared to KernelSHAP. We also show that our Dynamic
WindowSHAP algorithm focuses more on the most important time steps and provides
more understandable explanations. As a result, WindowSHAP not only accelerates
the calculation of Shapley values for time-series data, but also delivers more
understandable explanations with higher quality.
