<p>(Azoury and Warmuth, 2001) have shown that minimizing the KL-divergence between two exponential family distributions is the same as minimizing the Bregman divergence that corresponds to their cumulant function, i.e.</p>

<p>$KL(P_{F,\theta_1} || P_{F,\theta_2}) = D_F(\theta_1 || \theta_2)$</p>

<p>For $f = \nabla F$, and $F^*$ the Fenchel conjugate of $F$, we get the following nice relationship for Bregman divergences:</p>

<p>$D_F(\theta_1 || \theta_2) = D_{F^*}(f(\theta_2) || f(\theta_1))$</p>

<p>See Lemma 4 in <a href="http://webdocs.cs.ualberta.ca/~whitem/publications/12aistats-reverse.pdf" rel="nofollow">this paper</a>, which also has references to other work that discusses this connection. The above can also be written without the $f$ transfers on $\theta$. So, there is a simple relationship between these two losses, i.e. between minimizing with $\theta_1$ as the first argument or with $\theta_2$ as the first argument.</p>

<p>Also, I feel I should mention that we can generally talk about maximizing likelihood in terms of Bregman divergences and P(data | params), rather than KL-divergences on densities. The above linked paper shows that</p>

<p>$\max_\theta log \ p(data | \theta) = \min_\theta D_F(\theta | data)$ </p>

<p>The relationship above where we swapped the two variables of interest in the loss has a correspondence here:</p>

<p>$p(\theta | data) \ p(data) = p(data | \theta) \ p(\theta)$</p>

<p>giving</p>

<p>$log \ p(\theta | data) = log \ p(data | \theta) + log \ p(\theta) - log \ p(data)$</p>

<p>Since we typically do not put a prior on data, we find that maximizing the likelihood of $p(\theta | data)$ (typically called the MAP estimate) corresponds to maximizing likelihood of the data with a prior on the parameters. Correspondingly, maximum likelihood is like the MAP estimate, with a non-informative, uniform prior on $\theta$. </p>

<p>K. S. Azoury and M. K. Warmuth. Relative loss bounds for on-line density estimation with the exponential family of distributions. 2001</p>

CC BY-SA 3.0
2014-07-21T14:06:10.293Z
It has been a while since I asked the question and the answer is pretty dense, so it will take a while before I can get the spare cycles to understand it. I am writing this comment so that you will understand I am grateful for your answer and will be looking at it as soon as I can to accept it or ask for clarification.
Eponymous
CC BY-SA 3.0
2014-07-18T21:29:02.497Z
2014-08-07T22:10:41.467Z
2014-08-07T22:10:41.467Z
Martha White
<p>If $D_{KL}$ is the Kullback-Leibler divergence, minimizing $D_{KL}(P_{data}||P_{parameters})$ performs maximum likelihood estimation of the parameters. What happens if you minimize $D_{KL}(P_{parameters} || P_{data})$?</p>

CC BY-SA 3.0
2013-07-01T05:41:29.070Z
I'm not sure why this was voted down. It's a perfectly reasonable question.
Suresh Venkat
CC BY-SA 3.0
2013-07-02T22:42:18.840Z
You will get a set of parameter resulting in a distribution $P_{parameters}$, that, had it been the true distribution, would have made the cost of assuming $P_{data}$ instead of $P_{distribution}$ minimal. An example: suppose you want to estimate the frequency of each character 'A','B' and 'C' in a stream in order to compress it with a Huffman code. if the parameter p has value 1, then $Prob('A') = 0.5, Prob('B') = 0.45, Prob('C') = 0.05$, and if parameter p has value 0, then $Prob('A') =.5, Prob('B') =0.5, Prob('C') = 0$. Also, $P_{data}(A)=0.5, P_{data}(B)=0.499, P_{data}(C) = 0.001$
user8477
CC BY-SA 3.0
2013-07-02T22:51:30.680Z
Now it's clear that you should estimate the value of the parameter p to be 1, not 0, since you have 'C' characters in your data. But if you minimize $D_{KL} (P_{parameter}||P_{data})$ you will choose instead the value 0 for p.
user8477
CC BY-SA 3.0
2013-07-03T15:54:59.170Z
I should have thought of using a simple parameterized distribution like you provided. (Though I think your conclusion that $p$ ought to be 1 is not so clear if we make $P_{param}(C|p=0)=10^{-5}$). This gives me a good way to think about the question.
Eponymous
CC BY-SA 3.0
2013-07-03T16:45:34.540Z
It is interesting to note that your distribution can't be used for the MLE version of the minimization because $D_{KL}(P||Q)$ is undefined if there exists an $\alpha$ such that $Q(\alpha) = 0$ and $P(\alpha) \neq 0$. Since $P_{data}(C)=10^{-3}$ and $P_{param}(C|p=0)=0$ this makes $D_{KL}(P_{data}(\cdot)||P_{param}(\cdot|p=0))$ undefined.
Eponymous
CC BY-SA 3.0
2013-07-03T17:17:56.677Z
My previous observation brings up a subtle distinction between MLE and minimizing $D_{KL}(P_{data}||P_{param})$ - minimizing $D_{KL}$ is equivalent to maximizing the LOG likelihood not the likelihood. Usually these two are the same, but zeros complicate things. In the case of user8477's distribution, the likelihood exists, but the log likelihood does not exist because of zeros. Thus, you can do MLE but not the $D_{KL}$ analogue.
Eponymous
CC BY-SA 3.0
2013-07-03T19:03:56.510Z
Well, if as you said, you set $P_{param}(C|p=0)$ to a sufficiently small value instead of 0, then $D_{KL}(P_{data}(.)||P_{param}(.|p=0))$  will be well defined and arbitrarily large. When I gave my example I thought of the $D_{KL}$  as being "infinite" instead of undefined, but by choosing a sufficiently small value for $P_{param}(C|p=0)$ you can get around this problem.
user8477
<machine-learning><st.statistics>
What happens if you minimize $D_{KL}(P_{parameters} || P_{data})$ under the Kullback-Leibler divergence?
Eponymous
# What happens if you minimize $D_{KL}(P_{parameters} || P_{data})$ under the Kullback-Leibler divergence?

(Asked by: username_0 on 2013-06-30)

If $D\_{KL}$ is the Kullback-Leibler divergence, minimizing $D\_{KL}(P\_{data}||P\_{parameters})$ performs maximum likelihood estimation of the parameters. What happens if you minimize $D\_{KL}(P\_{parameters} || P\_{data})$?

**Tags:** machine-learning, st.statistics
> **Comments:**
> **username_1**: I'm not sure why this was voted down. It's a perfectly reasonable question.  
> **username_2**: You will get a set of parameter resulting in a distribution $P_{parameters}$, that, had it been the true distribution, would have made the cost of assuming $P_{data}$ instead of $P_{distribution}$ minimal. An example: suppose you want to estimate the frequency of each character 'A','B' and 'C' in a stream in order to compress it with a Huffman code. if the parameter p has value 1, then $Prob('A') = 0.5, Prob('B') = 0.45, Prob('C') = 0.05$, and if parameter p has value 0, then $Prob('A') =.5, Prob('B') =0.5, Prob('C') = 0$. Also, $P_{data}(A)=0.5, P_{data}(B)=0.499, P_{data}(C) = 0.001$  
> **username_2**: Now it's clear that you should estimate the value of the parameter p to be 1, not 0, since you have 'C' characters in your data. But if you minimize $D_{KL} (P_{parameter}||P_{data})$ you will choose instead the value 0 for p.  
> **username_0**: I should have thought of using a simple parameterized distribution like you provided. (Though I think your conclusion that $p$ ought to be 1 is not so clear if we make $P_{param}(C|p=0)=10^{-5}$). This gives me a good way to think about the question.  
> **username_0**: It is interesting to note that your distribution can't be used for the MLE version of the minimization because $D_{KL}(P||Q)$ is undefined if there exists an $\alpha$ such that $Q(\alpha) = 0$ and $P(\alpha) \neq 0$. Since $P_{data}(C)=10^{-3}$ and $P_{param}(C|p=0)=0$ this makes $D_{KL}(P_{data}(\cdot)||P_{param}(\cdot|p=0))$ undefined.  
> **username_0**: My previous observation brings up a subtle distinction between MLE and minimizing $D_{KL}(P_{data}||P_{param})$ - minimizing $D_{KL}$ is equivalent to maximizing the LOG likelihood not the likelihood. Usually these two are the same, but zeros complicate things. In the case of user8477's distribution, the likelihood exists, but the log likelihood does not exist because of zeros. Thus, you can do MLE but not the $D_{KL}$ analogue.  
> **username_2**: Well, if as you said, you set $P_{param}(C|p=0)$ to a sufficiently small value instead of 0, then $D_{KL}(P_{data}(.)||P_{param}(.|p=0))$  will be well defined and arbitrarily large. When I gave my example I thought of the $D_{KL}$  as being "infinite" instead of undefined, but by choosing a sufficiently small value for $P_{param}(C|p=0)$ you can get around this problem.  

---

## 1 answers

### Answer by username_3 on 2014-07-18. Score: 1 | Accepted answer

(Azoury and Warmuth, 2001) have shown that minimizing the KL-divergence between two exponential family distributions is the same as minimizing the Bregman divergence that corresponds to their cumulant function, i.e.

$KL(P\_{F,\theta\_1} || P\_{F,\theta\_2}) = D\_F(\theta\_1 || \theta\_2)$

For $f = \nabla F$, and $F^\*$ the Fenchel conjugate of $F$, we get the following nice relationship for Bregman divergences:

$D\_F(\theta\_1 || \theta\_2) = D\_{F^\*}(f(\theta\_2) || f(\theta\_1))$

See Lemma 4 in [this paper](http://webdocs.cs.ualberta.ca/~whitem/publications/12aistats-reverse.pdf), which also has references to other work that discusses this connection. The above can also be written without the $f$ transfers on $\theta$. So, there is a simple relationship between these two losses, i.e. between minimizing with $\theta\_1$ as the first argument or with $\theta\_2$ as the first argument.

Also, I feel I should mention that we can generally talk about maximizing likelihood in terms of Bregman divergences and P(data | params), rather than KL-divergences on densities. The above linked paper shows that

$\max\_\theta log \ p(data | \theta) = \min\_\theta D\_F(\theta | data)$

The relationship above where we swapped the two variables of interest in the loss has a correspondence here:

$p(\theta | data) \ p(data) = p(data | \theta) \ p(\theta)$

giving

$log \ p(\theta | data) = log \ p(data | \theta) + log \ p(\theta) - log \ p(data)$

Since we typically do not put a prior on data, we find that maximizing the likelihood of $p(\theta | data)$ (typically called the MAP estimate) corresponds to maximizing likelihood of the data with a prior on the parameters. Correspondingly, maximum likelihood is like the MAP estimate, with a non-informative, uniform prior on $\theta$.

K. S. Azoury and M. K. Warmuth. Relative loss bounds for on-line density estimation with the exponential family of distributions. 2001

> **Comments:**
> **username_0**: It has been a while since I asked the question and the answer is pretty dense, so it will take a while before I can get the spare cycles to understand it. I am writing this comment so that you will understand I am grateful for your answer and will be looking at it as soon as I can to accept it or ask for clarification.  

---

