  Spiking neural networks (SNNs) are posited as a biologically plausible
alternative to conventional neural architectures, with their core computational
framework resting on the extensively studied leaky integrate-and-fire (LIF)
neuron design. The stateful nature of LIF neurons has spurred ongoing
discussions about the ability of SNNs to process sequential data, akin to
recurrent neural networks (RNNs). Despite this, there remains a significant gap
in the exploration of current SNNs within the realm of long-range dependency
tasks. In this study, to extend the analysis of neuronal dynamics beyond
simplistic LIF mechanism, we present a novel class of stochastic spiking
neuronal model grounded in state space models. We expand beyond the scalar
hidden state representation of LIF neurons, which traditionally comprises only
the membrane potential, by proposing an n-dimensional hidden state.
Additionally, we enable fine-tuned formulation of neuronal dynamics across each
layer by introducing learnable parameters, as opposed to the fixed dynamics in
LIF neurons. We also develop a robust framework for scaling these neuronal
models to deep SNN-based architectures, ensuring efficient parallel training
while also adeptly addressing the challenge of non-differentiability of
stochastic spiking operation during the backward phase. Our models attain
state-of-the-art performance among SNN models across diverse long-range
dependency tasks, encompassing the Long Range Arena benchmark, permuted
sequential MNIST, and the Speech Command dataset. Moreover, we provide an
analysis of the energy efficiency advantages, emphasizing the sparse activity
pattern intrinsic to this spiking model.
