<p>I am reading this <a href="https://www.iacr.org/museum/shannon/shannon45.pdf" rel="nofollow noreferrer">classic paper</a> by Claude E. Shannon and I think there may be a couple of errors in his description of the properties of Entropy/Uncertainty. The screenshot shown at the bottom of this post is from page 9. Firstly, I believe there is a typo at the first red circle which should read <span class="math-container">$10^{50}$</span>. Secondly, I think the equation shown for the value of the entropy <span class="math-container">$H$</span> of this problem is also incorrect. I believe the correct equation for <span class="math-container">$H$</span> is derived as follows:</p>
<p>events <span class="math-container">$=\{p_1, p_2, ... , p_{10^{30}}, q_1, q_2, ..., q_{10^{50}}\}$</span></p>
<p>where events <span class="math-container">$p_i$</span> each have a probability <span class="math-container">$\frac{1}{2}(10^{-30})$</span> of occurring, and events <span class="math-container">$q_j$</span> each have a probability <span class="math-container">$\frac{1}{2}(10^{-50})$</span> of occuring</p>
<p><span class="math-container">\begin{align}
H &amp;= -\sum_{i=1}^{10^{30}} p_i\log{p_i} - \sum_{j=1}^{10^{50}} q_j\log{q_j} \\
&amp;= -\sum_{i=1}^{10^{30}} \frac{1}{2}(10^{-30})\log{(\frac{1}{2}(10^{-30}))} -\sum_{j=1}^{10^{50}} \frac{1}{2}(10^{-50})\log{(\frac{1}{2}(10^{-50}))} \\
&amp;= -\frac{1}{2}\log{(\frac{1}{2}(10^{-30}))} -\frac{1}{2}\log{(\frac{1}{2}(10^{-50}))} \\
&amp;\approx 40 
\end{align}</span></p>
<p>Hence, I think the equation for <span class="math-container">$H$</span> should read:</p>
<p><span class="math-container">$H = -\frac{1}{2}\log{\frac{1}{2}10^{-30}} -\frac{1}{2}\log{\frac{1}{2}10^{-50}}$</span></p>
<p>I would really appreciate if someone could let me know if I'm correct or where I'm wrong. I am not doubting Claude, just wondering if the document itself has typos. Thanks!</p>
<p><a href="https://i.sstatic.net/hCgeg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hCgeg.png" alt="page 9 part 8 of paper" /></a></p>

CC BY-SA 4.0
2021-03-22T09:57:41.377Z
Yes, you are right, these are typos. Btw, you can simplify $-\frac12\log(\frac1210^{-30})-\frac12\log(\frac1210^{-50})$ to $\frac12\log(2\cdot10^{30})+\frac12\log(2\cdot10^{50})=40+\log2$ (I assume from the context these are base-10 logarithms).
Emil Jeřábek
CC BY-SA 4.0
2021-03-22T11:01:03.230Z
Thanks, yes they're base-10. That's a cool simplification.
nuggimane
<it.information-theory><shannon-entropy><paper-review><cryptography>
Error in entropy properties in Mathematical Theory of Cryptography by Claude E. Shannon
nuggimane
# Error in entropy properties in Mathematical Theory of Cryptography by Claude E. Shannon

(Asked by: username_0 on 2021-03-22)

I am reading this [classic paper](https://www.iacr.org/museum/shannon/shannon45.pdf) by Claude E. Shannon and I think there may be a couple of errors in his description of the properties of Entropy/Uncertainty. The screenshot shown at the bottom of this post is from page 9. Firstly, I believe there is a typo at the first red circle which should read $10^{50}$. Secondly, I think the equation shown for the value of the entropy $H$ of this problem is also incorrect. I believe the correct equation for $H$ is derived as follows:

events $=\{p\_1, p\_2, ... , p\_{10^{30}}, q\_1, q\_2, ..., q\_{10^{50}}\}$

where events $p\_i$ each have a probability $\frac{1}{2}(10^{-30})$ of occurring, and events $q\_j$ each have a probability $\frac{1}{2}(10^{-50})$ of occuring

\begin{align}
H &= -\sum\_{i=1}^{10^{30}} p\_i\log{p\_i} - \sum\_{j=1}^{10^{50}} q\_j\log{q\_j} \\
&= -\sum\_{i=1}^{10^{30}} \frac{1}{2}(10^{-30})\log{(\frac{1}{2}(10^{-30}))} -\sum\_{j=1}^{10^{50}} \frac{1}{2}(10^{-50})\log{(\frac{1}{2}(10^{-50}))} \\
&= -\frac{1}{2}\log{(\frac{1}{2}(10^{-30}))} -\frac{1}{2}\log{(\frac{1}{2}(10^{-50}))} \\
&\approx 40
\end{align}

Hence, I think the equation for $H$ should read:

$H = -\frac{1}{2}\log{\frac{1}{2}10^{-30}} -\frac{1}{2}\log{\frac{1}{2}10^{-50}}$

I would really appreciate if someone could let me know if I'm correct or where I'm wrong. I am not doubting Claude, just wondering if the document itself has typos. Thanks!

[![page 9 part 8 of paper](https://i.sstatic.net/hCgeg.png)](https://i.sstatic.net/hCgeg.png)

**Tags:** it.information-theory, shannon-entropy, paper-review, cryptography

> **Comments:**
> **username_1**: Yes, you are right, these are typos. Btw, you can simplify $-\frac12\log(\frac1210^{-30})-\frac12\log(\frac1210^{-50})$ to $\frac12\log(2\cdot10^{30})+\frac12\log(2\cdot10^{50})=40+\log2$ (I assume from the context these are base-10 logarithms).  
> **username_0**: Thanks, yes they're base-10. That's a cool simplification.  

---

## 0 answers

