  In this paper, we study sampling from a posterior derived from a neural
network. We propose a new probabilistic model consisting of adding noise at
every pre- and post-activation in the network, arguing that the resulting
posterior can be sampled using an efficient Gibbs sampler. For small models,
the Gibbs sampler attains similar performances as the state-of-the-art Markov
chain Monte Carlo (MCMC) methods, such as the Hamiltonian Monte Carlo (HMC) or
the Metropolis adjusted Langevin algorithm (MALA), both on real and synthetic
data. By framing our analysis in the teacher-student setting, we introduce a
thermalization criterion that allows us to detect when an algorithm, when run
on data with synthetic labels, fails to sample from the posterior. The
criterion is based on the fact that in the teacher-student setting we can
initialize an algorithm directly at equilibrium.
