  The highly irregular spiking activity of cortical neurons and behavioral
variability suggest that the brain could operate in a fundamentally
probabilistic way. Mimicking how the brain implements and learns probabilistic
computation could be a key to developing machine intelligence that can think
more like humans. In this work, we propose a theory of stochastic neural
computing (SNC) in which streams of noisy inputs are transformed and processed
through populations of nonlinearly coupled spiking neurons. To account for the
propagation of correlated neural variability, we derive from first principles a
moment embedding for spiking neural network (SNN). This leads to a new class of
deep learning model called the moment neural network (MNN) which naturally
generalizes rate-based neural networks to second order. As the MNN faithfully
captures the stationary statistics of spiking neural activity, it can serve as
a powerful proxy for training SNN with zero free parameters. Through joint
manipulation of mean firing rate and noise correlations in a task-driven way,
the model is able to learn inference tasks while simultaneously minimizing
prediction uncertainty, resulting in enhanced inference speed. We further
demonstrate the application of our method to Intel's Loihi neuromorphic
hardware. The proposed theory of SNC may open up new opportunities for
developing machine intelligence capable of computing uncertainty and for
designing unconventional computing architectures.
