  This paper shows how sparse, high-dimensional probability distributions could
be represented by neurons with exponential compression. The representation is a
novel application of compressive sensing to sparse probability distributions
rather than to the usual sparse signals. The compressive measurements
correspond to expected values of nonlinear functions of the probabilistically
distributed variables. When these expected values are estimated by sampling,
the quality of the compressed representation is limited only by the quality of
sampling. Since the compression preserves the geometric structure of the space
of sparse probability distributions, probabilistic computation can be performed
in the compressed domain. Interestingly, functions satisfying the requirements
of compressive sensing can be implemented as simple perceptrons. If we use
perceptrons as a simple model of feedforward computation by neurons, these
results show that the mean activity of a relatively small number of neurons can
accurately represent a high-dimensional joint distribution implicitly, even
without accounting for any noise correlations. This comprises a novel
hypothesis for how neurons could encode probabilities in the brain.
