  Spiking neural networks (SNN) are able to learn spatiotemporal features while
using less energy, especially on neuromorphic hardware. The most widely used
spiking neuron in deep learning is the Leaky Integrate and Fire (LIF) neuron.
LIF neurons operate sequentially, however, since the computation of state at
time t relies on the state at time t-1 being computed. This limitation is
shared with Recurrent Neural Networks (RNN) and results in slow training on
Graphics Processing Units (GPU). In this paper, we propose the Stochastic
Parallelizable Spiking Neuron (SPSN) to overcome the sequential training
limitation of LIF neurons. By separating the linear integration component from
the non-linear spiking function, SPSN can be run in parallel over time. The
proposed approach results in performance comparable with the state-of-the-art
for feedforward neural networks on the Spiking Heidelberg Digits (SHD) dataset,
outperforming LIF networks while training 10 times faster and outperforming
non-spiking networks with the same network architecture. For longer input
sequences of 10000 time-steps, we show that the proposed approach results in
4000 times faster training, thus demonstrating the potential of the proposed
approach to accelerate SNN training for very large datasets.
