  We expand upon a natural analogy between Bayesian statistics and statistical
physics in which sample size corresponds to inverse temperature. This analogy
motivates the definition of two novel statistical quantities: a learning
capacity and a Gibbs entropy. The analysis of the learning capacity,
corresponding to the heat capacity in thermal physics, leads to new insight
into the mechanism of learning and explains why some models have
anomalously-high learning performance. We explore the properties of the
learning capacity in a number of examples, including a sloppy model. Next, we
propose that the Gibbs entropy provides a natural device for counting
distinguishable distributions in the context of Bayesian inference. We use this
device to define a generalized principle of indifference (GPI) in which every
distinguishable model is assigned equal a priori probability. This principle
results in a new solution to a long-standing problem in Bayesian inference: the
definition of an objective or uninformative prior. A key characteristic of this
new approach is that it can be applied to analyses where the model dimension is
unknown and circumvents the automatic rejection of higher-dimensional models in
Bayesian inference.
