  We recapitulate the Bayesian formulation of neural network based classifiers
and show that, while sampling from the posterior does indeed lead to better
generalisation than is obtained by standard optimisation of the cost function,
even better performance can in general be achieved by sampling finite
temperature ($T$) distributions derived from the posterior. Taking the example
of two different deep (3 hidden layers) classifiers for MNIST data, we find
quite different $T$ values to be appropriate in each case. In particular, for a
typical neural network classifier a clear minimum of the test error is observed
at $T>0$. This suggests an early stopping criterion for full batch simulated
annealing: cool until the average validation error starts to increase, then
revert to the parameters with the lowest validation error. As $T$ is increased
classifiers transition from accurate classifiers to classifiers that have
higher training error than assigning equal probability to each class. Efficient
studies of these temperature-induced effects are enabled using a
replica-exchange Hamiltonian Monte Carlo simulation technique. Finally, we show
how thermodynamic integration can be used to perform model selection for deep
neural networks. Similar to the Laplace approximation, this approach assumes
that the posterior is dominated by a single mode. Crucially, however, no
assumption is made about the shape of that mode and it is not required to
precisely compute and invert the Hessian.
