  Statistical inference more often than not involves models which are
non-linear in the parameters thus leading to non-Gaussian posteriors. Many
computational and analytical tools exist that can deal with non-Gaussian
distributions, and empirical Gaussianisation transforms can reduce the amount
of non-Gaussianity in a distribution. Alternatively, in this work, we employ
methods from information geometry. The latter formulates a set of probability
distributions for some given model as a manifold employing a Riemannian
structure, equipped with a metric, the Fisher information. In this framework we
study the differential geometrical meaning of non-Gaussianities in a higher
order Fisher approximation, and their respective transformation behaviour under
re-parameterisation, which corresponds to a chart transition on the statistical
manifold. While weak non-Gaussianities vanish in normal coordinates in a first
order approximation, one can in general not find transformations that discard
non-Gaussianities globally. As an application we consider the likelihood of the
supernovae distance-redshift relation in cosmology for the parameter pair
($\Omega_{\mathrm{m_0}}$, $w$). We demonstrate the connection between
confidence intervals and geodesic length and demonstrate how the Lie-derivative
along the degeneracy directions gives hints at possible isometries of the
Fisher metric.
