<p>This is only a partial answer. I might update it if I decide to look at the implementation.</p>
<blockquote>
<p>I understand this as being fixed network layers that don't get updated
in training. Am I mistaken?</p>
</blockquote>
<p>Note that the mutual information is being estimated <strong>after</strong> an epoch. <em>At this point</em> the weights of the layers are fixed. Hence, the output of each layer is deterministic.</p>

CC BY-SA 4.0
2023-07-07T23:28:22.647Z
2023-07-07T23:28:22.647Z
Nichlas L.R
<p>I am trying to understand the <a href="https://arxiv.org/abs/1705.02436" rel="nofollow noreferrer">Nonlinear Information Bottlecneck paper</a> along with <a href="https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py" rel="nofollow noreferrer">their implementation</a>, but I am confused as to what is actually being calculated in the Mutual information <span class="math-container">$(I(Y, M))$</span> between the Target (<span class="math-container">$Y$</span>) and the Features (<span class="math-container">$M$</span>).</p>
<p><span class="math-container">$$I(Y, M) = H(Y) - H(Y|M) $$</span></p>
<p>I was under the impression that in the Information Bottleneck method, we want to maximize <span class="math-container">$I(Y, M)$</span> (and minimize  <span class="math-container">$I(X, M)$</span>, but I won't look at that in this question). In a simple MNIST classification problem, this would translate to maximizing the mutual information between the class distribution (which is discrete and uniform), and the latent feature distribution (continuous).</p>
<p>However, in the example code, the conditional entropy (<span class="math-container">$H(Y|M)$</span>) seems to be calculated between the prediction <span class="math-container">$\hat{Y}$</span> and the target <span class="math-container">$Y$</span>, rather than the conditional entropy between the feature <span class="math-container">$M$</span> and the target <span class="math-container">$Y$</span> (<a href="https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py#L51" rel="nofollow noreferrer">see code here</a>).</p>
<p>I thought the goal was to produce informative features, not informative predictions.</p>
<p>They say as well in the paper :</p>
<blockquote>
<p>&quot;To implement the decoding map <span class="math-container">$P_{\phi}(y|m)$</span>, the bottleneck layer states are passed through several deterministic neural network layers with parameters <span class="math-container">$\phi$</span>.&quot; (Page 12)</p>
</blockquote>
<p>I understand this as being fixed network layers that don't get updated in training. Am I mistaken?   Because in the implementation, the decoding layers are trainable (<a href="https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py#L43" rel="nofollow noreferrer">see code here</a>).</p>
<p>This adds to my confusion because, in their implementation, it seems like they are calculating <span class="math-container">$I(Y, \hat{Y})$</span> (<em>which is essentially categorical cross entropy</em>), rather than <span class="math-container">$I(Y, M)$</span>.</p>
<p>What am I missing?</p>
<p>Thank you,</p>
<p>Liam</p>

CC BY-SA 4.0
2023-07-11T01:55:59.907Z
Cross-posted: https://cstheory.stackexchange.com/q/53039/5038, https://ai.stackexchange.com/q/41157/1794, https://stats.stackexchange.com/q/620630/2921.  Please [do not post the same question on multiple sites](https://meta.stackexchange.com/q/64068).
D.W.
<machine-learning><it.information-theory><lg.learning><shannon-entropy>
Information Bottleneck - Calculating the Mutual information between the Labels and the Features
Liam F-A
# Information Bottleneck - Calculating the Mutual information between the Labels and the Features

(Asked by: username_0 on 2023-07-05)

I am trying to understand the [Nonlinear Information Bottlecneck paper](https://arxiv.org/abs/1705.02436) along with [their implementation](https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py), but I am confused as to what is actually being calculated in the Mutual information $(I(Y, M))$ between the Target ($Y$) and the Features ($M$).

$$I(Y, M) = H(Y) - H(Y|M) $$

I was under the impression that in the Information Bottleneck method, we want to maximize $I(Y, M)$ (and minimize $I(X, M)$, but I won't look at that in this question). In a simple MNIST classification problem, this would translate to maximizing the mutual information between the class distribution (which is discrete and uniform), and the latent feature distribution (continuous).

However, in the example code, the conditional entropy ($H(Y|M)$) seems to be calculated between the prediction $\hat{Y}$ and the target $Y$, rather than the conditional entropy between the feature $M$ and the target $Y$ ([see code here](https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py#L51)).

I thought the goal was to produce informative features, not informative predictions.

They say as well in the paper :

> "To implement the decoding map $P\_{\phi}(y|m)$, the bottleneck layer states are passed through several deterministic neural network layers with parameters $\phi$." (Page 12)

I understand this as being fixed network layers that don't get updated in training. Am I mistaken? Because in the implementation, the decoding layers are trainable ([see code here](https://github.com/artemyk/nonlinearIB/blob/8913c1b75fa573c8d386516ba0fc52d9e9d088de/code/mnistdemo.py#L43)).

This adds to my confusion because, in their implementation, it seems like they are calculating $I(Y, \hat{Y})$ (*which is essentially categorical cross entropy*), rather than $I(Y, M)$.

What am I missing?

Thank you,

Liam

**Tags:** machine-learning, it.information-theory, lg.learning, shannon-entropy

> **Comments:**
> **username_1**: Cross-posted: https://cstheory.stackexchange.com/q/53039/5038, https://ai.stackexchange.com/q/41157/1794, https://stats.stackexchange.com/q/620630/2921.  Please [do not post the same question on multiple sites](https://meta.stackexchange.com/q/64068).  

---

## 1 answers

### Answer by username_2 on 2023-07-07. Score: 1

This is only a partial answer. I might update it if I decide to look at the implementation.

> I understand this as being fixed network layers that don't get updated
> in training. Am I mistaken?

Note that the mutual information is being estimated **after** an epoch. *At this point* the weights of the layers are fixed. Hence, the output of each layer is deterministic.


---

