<p>By the simple transformation <span class="math-container">$p(x_0, y) = p(y | x_0)p(x_0) $</span> your expression merely becomes <span class="math-container">$p(x_0) KL(p (y|x_0), p(y))$</span>, where <span class="math-container">$KL(p,q)$</span> is the Kullback-Leibler divergence, and is always positive. therefore, your expression is always positive.</p>

CC BY-SA 2.5
2010-12-22T17:13:20.240Z
Terrific, thanks!  I was fooling around with similar transformations but somehow missed the obvious.
Michael McGowan
CC BY-SA 2.5
2010-12-22T17:32:15.410Z
yes. this particular transformation turns out to be very handy.
Suresh Venkat
CC BY-SA 4.0
2010-12-22T17:03:06.570Z
2022-12-21T10:17:35.320Z
2022-12-21T10:17:35.320Z
Suresh Venkat
<p>Please forgive me if this is not the right Stack Exchange (I also posted it at <a href="https://stats.stackexchange.com/questions/5698/can-the-mutual-information-of-a-cell-be-negative">Cross Validated</a>).  Please also forgive me for inventing terms.</p>

<p>For discrete random variables X and Y, the mutual information of X and Y can be defined as follows:
$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} 
                 p(x,y) \log{ \left( \frac{p(x,y)}{p_1(x)\,p_2(y)}
                              \right) }, \,\!$</p>

<p>I will define the mutual information of a "cell" $x_0$ to be:
$CI(x_0,Y) = \sum_{y \in Y} p(x_0,y) \log{ \left( \frac{p(x_0,y)}{p_1(x_0)\,p_2(y)}
                              \right) }, \,\!$</p>

<p>I'm not sure if this quantity goes by another name.  Essentially I'm restricting focus to a single state of variable X (and then the full MI can be calculated by summing all the cell MIs).</p>

<p>My question: is it guaranteed that $CI(x_0,Y) \ge 0$?  We know $I(X;Y)\ge0$ and we know that the pointwise mutual information can be negative.  I feel like CI should be nonnegative and that I might be missing some obvious proof.</p>

<pr.probability><it.information-theory>
Can the mutual information of a "cell" be negative?
Michael McGowan
# Can the mutual information of a "cell" be negative?

(Asked by: username_0 on 2010-12-22)

Please forgive me if this is not the right Stack Exchange (I also posted it at [Cross Validated](https://stats.stackexchange.com/questions/5698/can-the-mutual-information-of-a-cell-be-negative)). Please also forgive me for inventing terms.

For discrete random variables X and Y, the mutual information of X and Y can be defined as follows:
$I(X;Y) = \sum\_{y \in Y} \sum\_{x \in X}
p(x,y) \log{ \left( \frac{p(x,y)}{p\_1(x)\,p\_2(y)}
\right) }, \,\!$

I will define the mutual information of a "cell" $x\_0$ to be:
$CI(x\_0,Y) = \sum\_{y \in Y} p(x\_0,y) \log{ \left( \frac{p(x\_0,y)}{p\_1(x\_0)\,p\_2(y)}
\right) }, \,\!$

I'm not sure if this quantity goes by another name. Essentially I'm restricting focus to a single state of variable X (and then the full MI can be calculated by summing all the cell MIs).

My question: is it guaranteed that $CI(x\_0,Y) \ge 0$? We know $I(X;Y)\ge0$ and we know that the pointwise mutual information can be negative. I feel like CI should be nonnegative and that I might be missing some obvious proof.

**Tags:** pr.probability, it.information-theory


---

## 1 answers

### Answer by username_1 on 2010-12-22. Score: 16 | Accepted answer

By the simple transformation $p(x\_0, y) = p(y | x\_0)p(x\_0) $ your expression merely becomes $p(x\_0) KL(p (y|x\_0), p(y))$, where $KL(p,q)$ is the Kullback-Leibler divergence, and is always positive. therefore, your expression is always positive.

> **Comments:**
> **username_0**: Terrific, thanks!  I was fooling around with similar transformations but somehow missed the obvious.  
> **username_1**: yes. this particular transformation turns out to be very handy.  

---

