  Most existing Spiking Neural Network (SNN) works state that SNNs may utilize
temporal information dynamics of spikes. However, an explicit analysis of
temporal information dynamics is still missing. In this paper, we ask several
important questions for providing a fundamental understanding of SNNs: What are
temporal information dynamics inside SNNs? How can we measure the temporal
information dynamics? How do the temporal information dynamics affect the
overall learning performance? To answer these questions, we estimate the Fisher
Information of the weights to measure the distribution of temporal information
during training in an empirical manner. Surprisingly, as training goes on,
Fisher information starts to concentrate in the early timesteps. After
training, we observe that information becomes highly concentrated in earlier
few timesteps, a phenomenon we refer to as temporal information concentration.
We observe that the temporal information concentration phenomenon is a common
learning feature of SNNs by conducting extensive experiments on various
configurations such as architecture, dataset, optimization strategy, time
constant, and timesteps. Furthermore, to reveal how temporal information
concentration affects the performance of SNNs, we design a loss function to
change the trend of temporal information. We find that temporal information
concentration is crucial to building a robust SNN but has little effect on
classification accuracy. Finally, we propose an efficient iterative pruning
method based on our observation on temporal information concentration. Code is
available at
https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.
