  Spiking Neural Networks (SNNs) are biologically realistic and practically
promising in low-power computation because of their event-driven mechanism.
Usually, the training of SNNs suffers accuracy loss on various tasks, yielding
an inferior performance compared with ANNs. A conversion scheme is proposed to
obtain competitive accuracy by mapping trained ANNs' parameters to SNNs with
the same structures. However, an enormous number of time steps are required for
these converted SNNs, thus losing the energy-efficient benefit. Utilizing both
the accuracy advantages of ANNs and the computing efficiency of SNNs, a novel
SNN training framework is proposed, namely layer-wise ANN-to-SNN knowledge
distillation (LaSNN). In order to achieve competitive accuracy and reduced
inference latency, LaSNN transfers the learning from a well-trained ANN to a
small SNN by distilling the knowledge other than converting the parameters of
ANN. The information gap between heterogeneous ANN and SNN is bridged by
introducing the attention scheme, the knowledge in an ANN is effectively
compressed and then efficiently transferred by utilizing our layer-wise
distillation paradigm. We conduct detailed experiments to demonstrate the
effectiveness, efficacy, and scalability of LaSNN on three benchmark data sets
(CIFAR-10, CIFAR-100, and Tiny ImageNet). We achieve competitive top-1 accuracy
compared to ANNs and 20x faster inference than converted SNNs with similar
performance. More importantly, LaSNN is dexterous and extensible that can be
effortlessly developed for SNNs with different architectures/depths and input
encoding methods, contributing to their potential development.
