  In sampling-based Bayesian models of brain function, neural activities are
assumed to be samples from probability distributions that the brain uses for
probabilistic computation. However, a comprehensive understanding of how
mechanistic models of neural dynamics can sample from arbitrary distributions
is still lacking. We use tools from functional analysis and stochastic
differential equations to explore the minimum architectural requirements for
$\textit{recurrent}$ neural circuits to sample from complex distributions. We
first consider the traditional sampling model consisting of a network of
neurons whose outputs directly represent the samples (sampler-only network). We
argue that synaptic current and firing-rate dynamics in the traditional model
have limited capacity to sample from a complex probability distribution. We
show that the firing rate dynamics of a recurrent neural circuit with a
separate set of output units can sample from an arbitrary probability
distribution. We call such circuits reservoir-sampler networks (RSNs). We
propose an efficient training procedure based on denoising score matching that
finds recurrent and output weights such that the RSN implements Langevin
sampling. We empirically demonstrate our model's ability to sample from several
complex data distributions using the proposed neural dynamics and discuss its
applicability to developing the next generation of sampling-based brain models.
