<p>The entropy refers to a set of symbols (a text in your case, or the set of words in a language). The self-information refers to a symbol in a set (a word in your case).</p>

<p>The information content of a text depends on how common the words in the text are wrt the global usage of those words.</p>

<p>E.g. "Next saturday morning" should have less self-information (as a named entity) than "Large Hadron Collider", each of the words should have less self information (except for large, maybe). However the entropy may be higher in the first case, there are not many Large Hadron "things".</p>

<p>Maybe Markov chains or some model that is better suited for probability in natural language would give you better (more significant and useful) results.</p>

CC BY-SA 3.0
2013-10-03T14:31:14.790Z
One minor point: the information content of a text also depends on the *order* of the words.
Peter Shor 
CC BY-SA 3.0
2013-10-03T14:35:59.850Z
@PeterShor yes, definitively, I was going to point that out with Markov chains, due to the conditional probability of a word given the previous one(s), but then I forgot.
Trylks
CC BY-SA 3.0
2013-10-03T14:14:16.477Z
2013-10-03T14:14:16.477Z
Trylks
<p>Self-information applies to an individual outcome, <span class="math-container">$x$</span>. It measures how surprising that specific outcome is.</p>
<p>The entropy of process <span class="math-container">$X$</span> is the average amount of Shannon self-information something coming out of <span class="math-container">$X$</span> will have, assuming that things come out according to the <span class="math-container">$P_X$</span> distribution. It's the <em>average</em> amount of surprise.</p>
<p>Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0, and the rest of the time it's evenly distributed among [1,2,3,4]. Thus, there is 1 bit of self-information when a 0 comes out, and 3 bits when a 1 comes out (or a 2, a 3, or a 4). We take the average, and <span class="math-container">$(1/2 \times 1 +1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3) = 2$</span> bits of entropy.</p>

CC BY-SA 4.0
2018-01-25T22:13:50.433Z
2021-02-05T00:10:35.487Z
2021-02-05T00:10:35.487Z
Post169
<p>Two mistakes from Post169:</p>
<ol>
<li>Post169 wrote : &quot;....Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0 .....&quot;</li>
</ol>
<p>Random means equal probabilities. There is no random number generator, that gives you half the times 0. This is a bias generator.</p>
<ol start="2">
<li>The equation that Post169 wrote is wrong. The result with this wrong equation is -2 (and not 2). The write equation is −[1/2×(-1) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3)] =2 bits.</li>
</ol>
<p>log(base2)0.5= -1, and log(base2)0.125= -3</p>

CC BY-SA 4.0
2021-02-05T04:01:06.530Z
"Random means equal probabilities. " $\leadsto$ no, it doesn't. That would be *uniformly* random; there exist probability distributions (such as the one in Post169's answer) which are not uniform over their domain.
Clement C.
CC BY-SA 4.0
2021-02-05T14:20:06.930Z
Hi @kraka, thank you for pointing things out. I've corrected the equation. 
While it is certainly the most random when all possibilities have equal probability, it is standard to refer to various random probability distribution functions.
BTW, next time you see an error in someone's answer and want to address it, it's best to add a comment below the answer itself, rather than creating a new answer to the question.
Post169
CC BY-SA 4.0
2021-02-25T01:02:38.390Z
To Clement C. : Of course there exist probability distributions which are not uniform over their domain, but in these distributions,  all outcomes are NOT equally likely. "Post169" clearly talk about "random number generator". There is not random number generator, that gives outcomes (numbers) with not equally probabilities. Do not confused by words. Here is Science, not Poetry.
kraka
CC BY-SA 4.0
2021-02-25T01:08:45.123Z
To Post169 : Thanks for your note, about add a comment than create a new answer. Really i did not know how to do this. I apologize.
kraka
CC BY-SA 4.0
2021-01-30T16:27:25.463Z
2021-01-30T17:12:12.200Z
2021-01-30T17:12:12.200Z
kraka
<p>I get a bit confused about different definitions of entropy and/or self-information.</p>

<p>Entropy?</p>

<p>$$ H(X) = - \sum_{x \in X} P_X(x) \cdot \log{\left(P_X(x)\right)} $$</p>

<p>Self-information?</p>

<p>$$ I(x) = - \log{P_X(x)} $$
$$ I(X) = - \sum_{x \in X} \log{\left(P_X(x)\right)} $$</p>

<p>When do I use which formulate to calculate what? Generally, I am interested in the average information content of a text. Here I would calculate the frequency of each word, the probability, sum up the negative logarithm of this probability and average it by the number of words.</p>

<p>What am I missing here?</p>

<it.information-theory><shannon-entropy>
Difference between self-information and entropy
Guarana Joe
# Difference between self-information and entropy

(Asked by: username_0 on 2013-10-03)

I get a bit confused about different definitions of entropy and/or self-information.

Entropy?

$$ H(X) = - \sum\_{x \in X} P\_X(x) \cdot \log{\left(P\_X(x)\right)} $$

Self-information?

$$ I(x) = - \log{P\_X(x)} $$
$$ I(X) = - \sum\_{x \in X} \log{\left(P\_X(x)\right)} $$

When do I use which formulate to calculate what? Generally, I am interested in the average information content of a text. Here I would calculate the frequency of each word, the probability, sum up the negative logarithm of this probability and average it by the number of words.

What am I missing here?

**Tags:** it.information-theory, shannon-entropy

---

## 3 answers

### Answer by username_1 on 2013-10-03. Score: 4 | Accepted answer

The entropy refers to a set of symbols (a text in your case, or the set of words in a language). The self-information refers to a symbol in a set (a word in your case).

The information content of a text depends on how common the words in the text are wrt the global usage of those words.

E.g. "Next saturday morning" should have less self-information (as a named entity) than "Large Hadron Collider", each of the words should have less self information (except for large, maybe). However the entropy may be higher in the first case, there are not many Large Hadron "things".

Maybe Markov chains or some model that is better suited for probability in natural language would give you better (more significant and useful) results.

> **Comments:**
> **username_2**: One minor point: the information content of a text also depends on the *order* of the words.  
> **username_1**: @PeterShor yes, definitively, I was going to point that out with Markov chains, due to the conditional probability of a word given the previous one(s), but then I forgot.  

---

### Answer by username_3 on 2018-01-25. Score: 4

Self-information applies to an individual outcome, $x$. It measures how surprising that specific outcome is.

The entropy of process $X$ is the average amount of Shannon self-information something coming out of $X$ will have, assuming that things come out according to the $P\_X$ distribution. It's the *average* amount of surprise.

Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0, and the rest of the time it's evenly distributed among [1,2,3,4]. Thus, there is 1 bit of self-information when a 0 comes out, and 3 bits when a 1 comes out (or a 2, a 3, or a 4). We take the average, and $(1/2 \times 1 +1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3) = 2$ bits of entropy.


---

### Answer by username_4 on 2021-01-30. Score: -2

Two mistakes from Post169:

1. Post169 wrote : "....Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0 ....."

Random means equal probabilities. There is no random number generator, that gives you half the times 0. This is a bias generator.

2. The equation that Post169 wrote is wrong. The result with this wrong equation is -2 (and not 2). The write equation is −[1/2×(-1) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3)] =2 bits.

log(base2)0.5= -1, and log(base2)0.125= -3

> **Comments:**
> **username_5**: "Random means equal probabilities. " $\leadsto$ no, it doesn't. That would be *uniformly* random; there exist probability distributions (such as the one in Post169's answer) which are not uniform over their domain.  
> **username_3**: Hi @kraka, thank you for pointing things out. I've corrected the equation.  While it is certainly the most random when all possibilities have equal probability, it is standard to refer to various random probability distribution functions. BTW, next time you see an error in someone's answer and want to address it, it's best to add a comment below the answer itself, rather than creating a new answer to the question.  
> **username_4**: To Clement C. : Of course there exist probability distributions which are not uniform over their domain, but in these distributions,  all outcomes are NOT equally likely. "Post169" clearly talk about "random number generator". There is not random number generator, that gives outcomes (numbers) with not equally probabilities. Do not confused by words. Here is Science, not Poetry.  
> **username_4**: To Post169 : Thanks for your note, about add a comment than create a new answer. Really i did not know how to do this. I apologize.  

---

