  Spiking Neural Networks (SNN) are energy-efficient computing architectures
that exchange spikes for processing information, unlike classical Artificial
Neural Networks (ANN). Due to this, SNNs are better suited for real-life
deployments. However, similar to ANNs, SNNs also benefit from deeper
architectures to obtain improved performance. Furthermore, like the deep ANNs,
the memory, compute and power requirements of SNNs also increase with model
size, and model compression becomes a necessity. Knowledge distillation is a
model compression technique that enables transferring the learning of a large
machine learning model to a smaller model with minimal loss in performance. In
this paper, we propose techniques for knowledge distillation in spiking neural
networks for the task of image classification. We present ways to distill
spikes from a larger SNN, also called the teacher network, to a smaller one,
also called the student network, while minimally impacting the classification
accuracy. We demonstrate the effectiveness of the proposed method with detailed
experiments on three standard datasets while proposing novel distillation
methodologies and loss functions. We also present a multi-stage knowledge
distillation technique for SNNs using an intermediate network to obtain higher
performance from the student network. Our approach is expected to open up new
avenues for deploying high performing large SNN models on resource-constrained
hardware platforms.
