  Sensory perception originates from the responses of sensory neurons, which
react to a collection of sensory signals linked to various physical attributes
of a singular perceptual object. Unraveling how the brain extracts perceptual
information from these neuronal responses is a pivotal challenge in both
computational neuroscience and machine learning. Here we introduce a
statistical mechanical theory, where perceptual information is first encoded in
the correlated variability of sensory neurons and then reformatted into the
firing rates of downstream neurons. Applying this theory, we illustrate the
encoding of motion direction using neural covariance and demonstrate
high-fidelity direction recovery by spiking neural networks. Networks trained
under this theory also show enhanced performance in classifying natural images,
achieving higher accuracy and faster inference speed. Our results challenge the
traditional view of neural covariance as a secondary factor in neural coding,
highlighting its potential influence on brain function.
