  In Bayesian statistics probability distributions express beliefs. However,
for many problems the beliefs cannot be computed analytically and
approximations of beliefs are needed. We seek a loss function that quantifies
how "embarrassing" it is to communicate a given approximation. We reproduce and
discuss an old proof showing that there is only one ranking under the
requirements that (1) the best ranked approximation is the non-approximated
belief and (2) that the ranking judges approximations only by their predictions
for actual outcomes. The loss function that is obtained in the derivation is
equal to the Kullback-Leibler divergence when normalized. This loss function is
frequently used in the literature. However, there seems to be confusion about
the correct order in which its functional arguments, the approximated and
non-approximated beliefs, should be used. The correct order ensures that the
recipient of a communication is only deprived of the minimal amount of
information. We hope that the elementary derivation settles the apparent
confusion. For example when approximating beliefs with Gaussian distributions
the optimal approximation is given by moment matching. This is in contrast to
many suggested computational schemes.
