  This paper concerns the approximation of probability measures on
$\mathbf{R}^d$ with respect to the Kullback-Leibler divergence. Given an
admissible target measure, we show the existence of the best approximation,
with respect to this divergence, from certain sets of Gaussian measures and
Gaussian mixtures. The asymptotic behavior of such best approximations is then
studied in the small parameter limit where the measure concentrates; this
asymptotic behaviour is characterized using $\Gamma$-convergence. The theory
developed is then applied to understanding the frequentist consistency of
Bayesian inverse problems. For a fixed realization of noise, we show the
asymptotic normality of the posterior measure in the small noise limit. Taking
into account the randomness of the noise, we prove a Bernstein-Von Mises type
result for the posterior measure.
