  Third-generation artificial neural networks, Spiking Neural Networks (SNNs),
can be efficiently implemented on hardware. Their implementation on
neuromorphic chips opens a broad range of applications, such as machine
learning-based autonomous control and intelligent biomedical devices. In
critical applications, however, insight into the reasoning of SNNs is
important, thus SNNs need to be equipped with the ability to explain how
decisions are reached. We present \textit{Temporal Spike Attribution} (TSA), a
local explanation method for SNNs. To compute the explanation, we aggregate all
information available in model-internal variables: spike times and model
weights. We evaluate TSA on artificial and real-world time series data and
measure explanation quality w.r.t. multiple quantitative criteria. We find that
TSA correctly identifies a small subset of input features relevant to the
decision (i.e., is output-complete and compact) and generates similar
explanations for similar inputs (i.e., is continuous). Further, our experiments
show that incorporating the notion of \emph{absent} spikes improves explanation
quality. Our work can serve as a starting point for explainable SNNs, with
future implementations on hardware yielding not only predictions but also
explanations in a broad range of application scenarios. Source code is
available at https://github.com/ElisaNguyen/tsa-explanations.
