  Interpreting deep learning time series models is crucial in understanding the
model's behavior and learning patterns from raw data for real-time
decision-making. However, the complexity inherent in transformer-based time
series models poses challenges in explaining the impact of individual features
on predictions. In this study, we leverage recent local interpretation methods
to interpret state-of-the-art time series models. To use real-world datasets,
we collected three years of daily case data for 3,142 US counties. Firstly, we
compare six transformer-based models and choose the best prediction model for
COVID-19 infection. Using 13 input features from the last two weeks, we can
predict the cases for the next two weeks. Secondly, we present an innovative
way to evaluate the prediction sensitivity to 8 population age groups over
highly dynamic multivariate infection data. Thirdly, we compare our proposed
perturbation-based interpretation method with related work, including a total
of eight local interpretation methods. Finally, we apply our framework to
traffic and electricity datasets, demonstrating that our approach is generic
and can be applied to other time-series domains.
