  In many applications in biology, engineering and economics, identifying
similarities and differences between distributions of data from complex
processes requires comparing finite categorical samples of discrete counts.
Statistical divergences quantify the difference between two distributions.
However, their estimation is very difficult and empirical methods often fail,
especially when the samples are small. We develop a Bayesian estimator of the
Kullback-Leibler divergence between two probability distributions that makes
use of a mixture of Dirichlet priors on the distributions being compared. We
study the properties of the estimator on two examples: probabilities drawn from
Dirichlet distributions, and random strings of letters drawn from Markov
chains. We extend the approach to the squared Hellinger divergence. Both
estimators outperform other estimation techniques, with better results for data
with a large number of categories and for higher values of divergences.
