  Communication by rare, binary spikes is a key factor for the energy
efficiency of biological brains. However, it is harder to train
biologically-inspired spiking neural networks (SNNs) than artificial neural
networks (ANNs). This is puzzling given that theoretical results provide exact
mapping algorithms from ANNs to SNNs with time-to-first-spike (TTFS) coding. In
this paper we analyze in theory and simulation the learning dynamics of
TTFS-networks and identify a specific instance of the vanishing-or-exploding
gradient problem. While two choices of SNN mappings solve this problem at
initialization, only the one with a constant slope of the neuron membrane
potential at threshold guarantees the equivalence of the training trajectory
between SNNs and ANNs with rectified linear units. We demonstrate that training
deep SNN models achieves the exact same performance as that of ANNs, surpassing
previous SNNs on image classification datasets such as MNIST/Fashion-MNIST,
CIFAR10/CIFAR100 and PLACES365. Our SNN accomplishes high-performance
classification with less than 0.3 spikes per neuron, lending itself for an
energy-efficient implementation. We show that fine-tuning SNNs with our robust
gradient descent algorithm enables their optimization for hardware
implementations with low latency and resilience to noise and quantization.
