  We propose a new concept of codivergence, which quantifies the similarity
between two probability measures $P_1, P_2$ relative to a reference probability
measure $P_0$. In the neighborhood of the reference measure $P_0$, a
codivergence behaves like an inner product between the measures $P_1 - P_0$ and
$P_2 - P_0$. Codivergences of covariance-type and correlation-type are
introduced and studied with a focus on two specific correlation-type
codivergences, the $\chi^2$-codivergence and the Hellinger codivergence. We
derive explicit expressions for several common parametric families of
probability distributions. For a codivergence, we introduce moreover the
divergence matrix as an analogue of the Gram matrix. It is shown that the
$\chi^2$-divergence matrix satisfies a data-processing inequality.
