  In this paper we leverage on probability over Riemannian manifolds to rethink
the interpretation of priors and posteriors in Bayesian inference. The main
mindshift is to move away from the idea that "a prior distribution establishes
a probability distribution over the parameters of our model" to the idea that
"a prior distribution establishes a probability distribution over probability
distributions". To do that we assume that our probabilistic model is a
Riemannian manifold with the Fisher metric. Under this mindset, any
distribution over probability distributions should be "intrinsic", that is,
invariant to the specific parametrization which is selected for the manifold.
We exemplify our ideas through a simple analysis of distributions over the
manifold of Bernoulli distributions. One of the major shortcomings of maximum a
posteriori estimates is that they depend on the parametrization. Based on the
understanding developed here, we can define the maximum a posteriori estimate
which is independent of the parametrization.
