<p>I was wondering about the self-information, the information content . If I have data and I measure different words in it, their probability and take the average mean of that, what is the lowest and what is the biggest outcome?</p>

<p>$$I(data) = \frac{\sum_{i=1}^{n} - log(P(word_i))}{n}$$ </p>

<p>The summed up probability is one, but how does that change with the logarithm? It seems, if one word is very unique and another word takes up the rest of the data in a big quantity, than the information content is higher:</p>

<p>$$\frac{(-\log 0.9999) + (-\log 0.0001)}{2} &gt; \frac{(-\log 0.9) + (-\log 0.1)}{2}$$</p>

<p>Does that mean, if one of the words limits to zero, that the information content becomes infinite? Does that make sense, seems strange.</p>

CC BY-SA 3.0
2013-10-01T20:11:37.650Z
If the frequency is actually zero, then that word should simply not be counted. But as the frequency of a word becomes very small, its information content does become unboundedly large. This makes sense if you view the information from word_i as the amount of "surprise" you would have if you saw word_i after drawing randomly from your data. It also makes sense if you view the information from word_i as the number of bits needed to specify its frequency. If its frequency is very small, say $2^{-30}$, then you need very many (in this case, 30) bits to specify it.
Joshua Grochow
CC BY-SA 3.0
2013-10-02T19:06:51.350Z
I don't think this is the standard definition of the information content, which is the *weighted* average of the self-information of the events, that is, the entropy: $I(\mathrm{data}) = \frac{1}{n} \sum -P(\mathrm{word}_i) \log P(\mathrm{word}_i).$ The quantity you are asking about can certainly be infinite.
Peter Shor 
CC BY-SA 3.0
2013-10-02T19:10:08.733Z
I didn't even notice - my previous comment applies to the standard definition of information content as in Peter Shor's comment.
Joshua Grochow
CC BY-SA 3.0
2013-10-03T11:40:43.407Z
@JoshuaGrochow Mh, so I only meant self-information here and for self-information I find the [definition](http://en.wikipedia.org/wiki/Self-information) $- \log{P(word_i)}$. The definition you have seems to be the averaged entropy?
Guarana Joe
CC BY-SA 3.0
2013-10-03T14:27:59.857Z
The *self-information* for a single word is indeed $-\log P(\mathrm{word}_i)$. The *entropy* is the *weighted* average of the self-information over all words. What you have defined is the ***average*** self-information over all words. I don't know what this quantity if used for, but it is certainly not the information content, as you seem to think it is. If you take some *samples* from a distribution, and take the average self-information of this, it does indeed correspond to an information content, but it also converges to the entropy as you take more and more samples.
Peter Shor 
CC BY-SA 3.0
2013-10-04T12:09:17.930Z
@PeterShor I see, ugh. I am sorry for the confusion of definitions. I found this definition in a dissertation and thought it's sound. It ought to compute the *information content* by applying different methods, dependent on the type of objective. Categorical values, numbers? Then the self-information. Text? Then the Term Frequency-Inverse Document Frequency, but in the end, all values are averaged over the total number of elements/objectives. Makes me really wonder, because this formular is not used anywhere else..
Guarana Joe
<it.information-theory><shannon-entropy>
Can the self-information be infinite?
Guarana Joe
# Can the self-information be infinite?

(Asked by: username_0 on 2013-10-01)

I was wondering about the self-information, the information content . If I have data and I measure different words in it, their probability and take the average mean of that, what is the lowest and what is the biggest outcome?

$$I(data) = \frac{\sum\_{i=1}^{n} - log(P(word\_i))}{n}$$

The summed up probability is one, but how does that change with the logarithm? It seems, if one word is very unique and another word takes up the rest of the data in a big quantity, than the information content is higher:

$$\frac{(-\log 0.9999) + (-\log 0.0001)}{2} > \frac{(-\log 0.9) + (-\log 0.1)}{2}$$

Does that mean, if one of the words limits to zero, that the information content becomes infinite? Does that make sense, seems strange.

**Tags:** it.information-theory, shannon-entropy
> **Comments:**
> **username_1**: If the frequency is actually zero, then that word should simply not be counted. But as the frequency of a word becomes very small, its information content does become unboundedly large. This makes sense if you view the information from word_i as the amount of "surprise" you would have if you saw word_i after drawing randomly from your data. It also makes sense if you view the information from word_i as the number of bits needed to specify its frequency. If its frequency is very small, say $2^{-30}$, then you need very many (in this case, 30) bits to specify it.  
> **username_2**: I don't think this is the standard definition of the information content, which is the *weighted* average of the self-information of the events, that is, the entropy: $I(\mathrm{data}) = \frac{1}{n} \sum -P(\mathrm{word}_i) \log P(\mathrm{word}_i).$ The quantity you are asking about can certainly be infinite.  
> **username_1**: I didn't even notice - my previous comment applies to the standard definition of information content as in Peter Shor's comment.  
> **username_0**: @JoshuaGrochow Mh, so I only meant self-information here and for self-information I find the [definition](http://en.wikipedia.org/wiki/Self-information) $- \log{P(word_i)}$. The definition you have seems to be the averaged entropy?  
> **username_2**: The *self-information* for a single word is indeed $-\log P(\mathrm{word}_i)$. The *entropy* is the *weighted* average of the self-information over all words. What you have defined is the ***average*** self-information over all words. I don't know what this quantity if used for, but it is certainly not the information content, as you seem to think it is. If you take some *samples* from a distribution, and take the average self-information of this, it does indeed correspond to an information content, but it also converges to the entropy as you take more and more samples.  
> **username_0**: @PeterShor I see, ugh. I am sorry for the confusion of definitions. I found this definition in a dissertation and thought it's sound. It ought to compute the *information content* by applying different methods, dependent on the type of objective. Categorical values, numbers? Then the self-information. Text? Then the Term Frequency-Inverse Document Frequency, but in the end, all values are averaged over the total number of elements/objectives. Makes me really wonder, because this formular is not used anywhere else..  

---

## 0 answers

