  Vanilla spiking neurons in Spiking Neural Networks (SNNs) use
charge-fire-reset neuronal dynamics, which can only be simulated serially and
can hardly learn long-time dependencies. We find that when removing reset, the
neuronal dynamics can be reformulated in a non-iterative form and parallelized.
By rewriting neuronal dynamics without reset to a general formulation, we
propose the Parallel Spiking Neuron (PSN), which generates hidden states that
are independent of their predecessors, resulting in parallelizable neuronal
dynamics and extremely high simulation speed. The weights of inputs in the PSN
are fully connected, which maximizes the utilization of temporal information.
To avoid the use of future inputs for step-by-step inference, the weights of
the PSN can be masked, resulting in the masked PSN. By sharing weights across
time-steps based on the masked PSN, the sliding PSN is proposed to handle
sequences of varying lengths. We evaluate the PSN family on simulation speed
and temporal/static data classification, and the results show the overwhelming
advantage of the PSN family in efficiency and accuracy. To the best of our
knowledge, this is the first study about parallelizing spiking neurons and can
be a cornerstone for the spiking deep learning research. Our codes are
available at \url{https://github.com/fangwei123456/Parallel-Spiking-Neuron}.
