  Kullback-Leibler (KL) divergence is one of the most important divergence
measures between probability distributions. In this paper, we prove several
properties of KL divergence between multivariate Gaussian distributions. First,
for any two $n$-dimensional Gaussian distributions $\mathcal{N}_1$ and
$\mathcal{N}_2$, we give the supremum of $KL(\mathcal{N}_1||\mathcal{N}_2)$
when $KL(\mathcal{N}_2||\mathcal{N}_1)\leq \varepsilon\ (\varepsilon>0)$. For
small $\varepsilon$, we show that the supremum is $\varepsilon +
2\varepsilon^{1.5} + O(\varepsilon^2)$. This quantifies the approximate
symmetry of small KL divergence between Gaussians. We also find the infimum of
$KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\geq
M\ (M>0)$. We give the conditions when the supremum and infimum can be
attained. Second, for any three $n$-dimensional Gaussians $\mathcal{N}_1$,
$\mathcal{N}_2$, and $\mathcal{N}_3$, we find an upper bound of
$KL(\mathcal{N}_1||\mathcal{N}_3)$ if $KL(\mathcal{N}_1||\mathcal{N}_2)\leq
\varepsilon_1$ and $KL(\mathcal{N}_2||\mathcal{N}_3)\leq \varepsilon_2$ for
$\varepsilon_1,\varepsilon_2\ge 0$. For small $\varepsilon_1$ and
$\varepsilon_2$, we show the upper bound is
$3\varepsilon_1+3\varepsilon_2+2\sqrt{\varepsilon_1\varepsilon_2}+o(\varepsilon_1)+o(\varepsilon_2)$.
This reveals that KL divergence between Gaussians follows a relaxed triangle
inequality. Importantly, all the bounds in the theorems presented in this paper
are independent of the dimension $n$. Finally, We discuss the applications of
our theorems in explaining counterintuitive phenomenon of flow-based model,
deriving deep anomaly detection algorithm, and extending one-step robustness
guarantee to multiple steps in safe reinforcement learning.
