  Spiking neural network (SNN) is studied in multidisciplinary domains to (i)
enable order-of-magnitudes energy-efficient AI inference and (ii)
computationally simulate neuro-scientific mechanisms. The lack of discrete
theory obstructs the practical application of SNN by limiting its performance
and nonlinearity support. We present a new optimization-theoretic perspective
of the discrete dynamics of spiking neurons. We prove that a discrete dynamical
system of simple integrate-and-fire models approximates the sub-gradient method
over unconstrained optimization problems. We practically extend our theory to
introduce a novel sign gradient descent (signGD)-based neuronal dynamics that
can (i) approximate diverse nonlinearities beyond ReLU and (ii) advance
ANN-to-SNN conversion performance in low time steps. Experiments on large-scale
datasets show that our technique achieves (i) state-of-the-art performance in
ANN-to-SNN conversion and (ii) is the first to convert new DNN architectures,
e.g., ConvNext, MLP-Mixer, and ResMLP. We publicly share our source code at
https://github.com/snuhcs/snn_signgd .
