<p>The way you are doing is not quite correct. Take a look at these tutorials 
 -  <a href="http://homepages.inf.ed.ac.uk/csutton/publications/ep-tutorial.pdf" rel="nofollow">http://homepages.inf.ed.ac.uk/csutton/publications/ep-tutorial.pdf</a> 
 -  <a href="http://web.engr.illinois.edu/~khashab2/learn/ep.pdf" rel="nofollow">http://web.engr.illinois.edu/~khashab2/learn/ep.pdf</a> </p>

<p>Go over the algorithms which show the EP procedures. Also note the way they explain the projection of the distributions using KL projection. </p>

CC BY-SA 3.0
2013-11-07T06:16:19.600Z
Hi. I posted this question when I was so junior. Actually, IMO, the better tutorial is in Chapter 10 of PRML (Bishop 2006). After that, the slide from Cambridge may be the best illustration http://mlg.eng.cam.ac.uk/teaching/4f13/1213/lect0809.pdf. Anyway, thanks for your answer.
tndoan
CC BY-SA 3.0
2013-11-07T02:58:11.187Z
2013-11-07T02:58:11.187Z
Daniel
<p>Expectation Propagation(EP) is now quite a standard technique to approximate marginal in graphical model. Moreover, EP can replace sum-product algorithm in factor graph. For this reason, I try to understand it by using very simple model but because of my limitation of knowledge, the result could not be reached. So would you please revise my progress to show me my errors?</p>

<p>My problem is simple. $x_1$ is the variable which is followed Gaussian distribution with the mean $\alpha$ and variance $\beta$. $x_2$ is the variable which satisfies $x_1 = x_2$. So the factor graph can be represented by </p>

<p>Gaussian ---$x_1$ ---- $f(x_1, x_2) $--- $x_2$</p>

<p>f is the factor function in factor graph and it is Dirac delta function. </p>

<p>$f(x_1, x_2) = \delta(x_1 - x_2)$</p>

<p>Based on the properties of factor graph, we have:</p>

<p>$p(x_1, x_2) = g(x_1) f(x_1, x_2)$</p>

<p>with $g(x_1)$ is the Gaussian distribution. When we take the integral over the variable $x_1$ of $p(x_1, x_2)$, we get that the marginal of $x_2$ is the Gaussian distribution with the same parameters with $x_1$. </p>

<p>However, I want to use EP technique to solve this problem. First of all, I will separate $f$ here like in Tom Minka's thesis</p>

<p>Gaussian ----- $x_1$ ------ $\tilde{f}_1(x_1)$  $\tilde{f}_2(x_2)$ ----- $x_2$</p>

<p>Now, I want to find the function $q(x)$ that is</p>

<p>$q(x_1, x_2) = g(x_1) \tilde{f}_1(x_1) \tilde{f}_2(x_2)$</p>

<p>$q$ is the approximate function of $p(x_1, x_2)$</p>

<p>I want to refine the function $\tilde{f}(x_1, x_2) = \tilde{f}_1(x_1)\tilde{f}_2(x_2)$, environment without $f$ is calculated</p>

<p>$q^{\f}(x_1, x_2) = g(x_1)$</p>

<p>so</p>

<p>$\hat{p}(x_1, x_2) = q^{\f}(x_1, x_2) f(x_1, x_2) = g(x_1) f(x_1, x_2)$</p>

<p>$\hat{p}(x_1) = g(x_1) \int_{x_2} f(x_1, x_2) dx_2$</p>

<p>$\hat{p}(x_2) = \int_{x_1} f(x_1, x_2) dx_1$</p>

<p>$q^{new} (x_1, x_2) = \hat{p}(x_1) \hat{p}(x_2)$</p>

<p>then</p>

<p>$\tilde{f}(x_1, x_2) = \frac{q^{new}(x_1, x_2)}{q^{\f}} = \int_{x_1} f(x_1, x_2) dx_1 \int_{x_2} f(x_1, x_2) dx_2$</p>

<p>so</p>

<p>$\tilde{f}_1(x_1) = \int_{x_2} f(x_1, x_2) dx_2$</p>

<p>$\tilde{f}_2(x_2) = \int_{x_1} f(x_1, x_2) dx_1$</p>

<p>Then I guess $\tilde{f}_1$ and $\tilde{f}_2$ are convergence, so</p>

<p>$q(x_1, x_2) = g(x_1) \int_{x_1} f(x_1, x_2) dx_1 \int_{x_2} f(x_1, x_2) dx_2$</p>

<p>Hence, I get stuck at this point. Would you please show me, what wrong did I do?</p>

<p>Thank you and looking forward to seeing your helps</p>

<machine-learning>
Applying Expectation Propagation to Factor Graph
tndoan
# Applying Expectation Propagation to Factor Graph

(Asked by: username_0 on 2013-02-11)

Expectation Propagation(EP) is now quite a standard technique to approximate marginal in graphical model. Moreover, EP can replace sum-product algorithm in factor graph. For this reason, I try to understand it by using very simple model but because of my limitation of knowledge, the result could not be reached. So would you please revise my progress to show me my errors?

My problem is simple. $x\_1$ is the variable which is followed Gaussian distribution with the mean $\alpha$ and variance $\beta$. $x\_2$ is the variable which satisfies $x\_1 = x\_2$. So the factor graph can be represented by

Gaussian ---$x\_1$ ---- $f(x\_1, x\_2) $--- $x\_2$

f is the factor function in factor graph and it is Dirac delta function.

$f(x\_1, x\_2) = \delta(x\_1 - x\_2)$

Based on the properties of factor graph, we have:

$p(x\_1, x\_2) = g(x\_1) f(x\_1, x\_2)$

with $g(x\_1)$ is the Gaussian distribution. When we take the integral over the variable $x\_1$ of $p(x\_1, x\_2)$, we get that the marginal of $x\_2$ is the Gaussian distribution with the same parameters with $x\_1$.

However, I want to use EP technique to solve this problem. First of all, I will separate $f$ here like in Tom Minka's thesis

Gaussian ----- $x\_1$ ------ $\tilde{f}\_1(x\_1)$ $\tilde{f}\_2(x\_2)$ ----- $x\_2$

Now, I want to find the function $q(x)$ that is

$q(x\_1, x\_2) = g(x\_1) \tilde{f}\_1(x\_1) \tilde{f}\_2(x\_2)$

$q$ is the approximate function of $p(x\_1, x\_2)$

I want to refine the function $\tilde{f}(x\_1, x\_2) = \tilde{f}\_1(x\_1)\tilde{f}\_2(x\_2)$, environment without $f$ is calculated

$q^{\f}(x\_1, x\_2) = g(x\_1)$

so

$\hat{p}(x\_1, x\_2) = q^{\f}(x\_1, x\_2) f(x\_1, x\_2) = g(x\_1) f(x\_1, x\_2)$

$\hat{p}(x\_1) = g(x\_1) \int\_{x\_2} f(x\_1, x\_2) dx\_2$

$\hat{p}(x\_2) = \int\_{x\_1} f(x\_1, x\_2) dx\_1$

$q^{new} (x\_1, x\_2) = \hat{p}(x\_1) \hat{p}(x\_2)$

then

$\tilde{f}(x\_1, x\_2) = \frac{q^{new}(x\_1, x\_2)}{q^{\f}} = \int\_{x\_1} f(x\_1, x\_2) dx\_1 \int\_{x\_2} f(x\_1, x\_2) dx\_2$

so

$\tilde{f}\_1(x\_1) = \int\_{x\_2} f(x\_1, x\_2) dx\_2$

$\tilde{f}\_2(x\_2) = \int\_{x\_1} f(x\_1, x\_2) dx\_1$

Then I guess $\tilde{f}\_1$ and $\tilde{f}\_2$ are convergence, so

$q(x\_1, x\_2) = g(x\_1) \int\_{x\_1} f(x\_1, x\_2) dx\_1 \int\_{x\_2} f(x\_1, x\_2) dx\_2$

Hence, I get stuck at this point. Would you please show me, what wrong did I do?

Thank you and looking forward to seeing your helps

**Tags:** machine-learning

---

## 1 answers

### Answer by username_1 on 2013-11-07. Score: 2

The way you are doing is not quite correct. Take a look at these tutorials
- <http://homepages.inf.ed.ac.uk/csutton/publications/ep-tutorial.pdf>
- <http://web.engr.illinois.edu/~khashab2/learn/ep.pdf>

Go over the algorithms which show the EP procedures. Also note the way they explain the projection of the distributions using KL projection.

> **Comments:**
> **username_0**: Hi. I posted this question when I was so junior. Actually, IMO, the better tutorial is in Chapter 10 of PRML (Bishop 2006). After that, the slide from Cambridge may be the best illustration http://mlg.eng.cam.ac.uk/teaching/4f13/1213/lect0809.pdf. Anyway, thanks for your answer.  

---

