  Computation using brain-inspired spiking neural networks (SNNs) with
neuromorphic hardware may offer orders of magnitude higher energy efficiency
compared to the current analog neural networks (ANNs). Unfortunately, training
SNNs with the same number of layers as state of the art ANNs remains a
challenge. To our knowledge the only method which is successful in this regard
is supervised training of ANN and then converting it to SNN. In this work we
directly train deep SNNs using backpropagation with surrogate gradient and find
that due to implicitly recurrent nature of feed forward SNN's the exploding or
vanishing gradient problem severely hinders their training. We show that this
problem can be solved by tuning the surrogate gradient function. We also
propose using batch normalization from ANN literature on input currents of SNN
neurons. Using these improvements we show that is is possible to train SNN with
ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets.
The trained SNN falls behind in accuracy compared to analogous ANN but requires
several orders of magnitude less inference time steps (as low as 10) to reach
good accuracy compared to SNNs obtained by conversion from ANN which require on
the order of 1000 time steps.
