<p>So definition of self-information is chosen to be as follows (shamelessly copied from wiki page of &quot;Information content&quot; ) :</p>
<ol>
<li><p>An event with probability 100% is perfectly unsurprising and yields no information.</p>
</li>
<li><p>The less probable an event is, the more surprising it is and the more information it yields.</p>
</li>
<li><p>If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.</p>
</li>
</ol>
<p>Point 3 of the definition answers your question.</p>
<p>If <span class="math-container">$A$</span> and <span class="math-container">$B$</span> independent events and C is an event where both <span class="math-container">$A$</span> and <span class="math-container">$B$</span> take place then
<span class="math-container">$$P(C) = P(A \cap B) = P(A)\cdot P(B)$$</span>
On the other hand information from <span class="math-container">$C$</span> is defined as follows:
<span class="math-container">$$I(C) = I(A)+I(B)$$</span>
Let <span class="math-container">$f(\cdot)$</span> be the functional form of the information expression, formally <span class="math-container">$$I(E) = f(P(E))$$</span>
Therefore,
<span class="math-container">$$f(P(C)) = f(P(A)) + f(P(B))$$</span>
<span class="math-container">$$ = f(P(A).P(B))$$</span></p>
<p>Can you find an <span class="math-container">$f(x)$</span> which is of the form <span class="math-container">$k\cdot x$</span>, such that the above properties are satisfied ? I think the answer is no.</p>
<p>But I would like to add that I don't recall very neat reasons for why it has to be only <span class="math-container">$k\log(x)$</span> as well. i.e why only <span class="math-container">$f(x) = k\log x $</span> satisfies <span class="math-container">$f(x\cdot y) = f(x)+f(y)$</span></p>

CC BY-SA 4.0
2020-12-29T17:58:22.103Z
For why its only Logx 
 --> https://www.reddit.com/r/math/comments/2x7an2/other_solutions_for_fxy_fx_fy/
SagarM
CC BY-SA 4.0
2020-12-29T17:39:07.173Z
2020-12-29T18:09:28.820Z
2020-12-29T18:09:28.820Z
SagarM
<p>I am completely new to information theory.
I was learning about information content but couldn’t make sense of why the relationship between information content and probability isn’t linear? And why it is sub-linear?
As the formula goes,</p>
<p><span class="math-container">$$
I(E) = -\log p(E).
$$</span></p>

CC BY-SA 4.0
2020-12-29T15:41:06.487Z
Why do you think the relationship between information content and probability should be linear?  The formula for the information in a random variable is $\sum_x p_x I(x)$, where $p_x$ is the probability of outcome $x$. If a random variable takes on $2^k$ values with equal probability, the information in this random variable is $k$ bits, and from this we can deduce that $I(x) = - \log_2(p_x)$.
Peter Shor 
<it.information-theory><shannon-entropy><information>
Why isn’t information-probability relationship linear?
Aether
# Why isn’t information-probability relationship linear?

(Asked by: username_0 on 2020-12-29)

I am completely new to information theory.
I was learning about information content but couldn’t make sense of why the relationship between information content and probability isn’t linear? And why it is sub-linear?
As the formula goes,

$$
I(E) = -\log p(E).
$$

**Tags:** it.information-theory, shannon-entropy, information
> **Comments:**
> **username_1**: Why do you think the relationship between information content and probability should be linear?  The formula for the information in a random variable is $\sum_x p_x I(x)$, where $p_x$ is the probability of outcome $x$. If a random variable takes on $2^k$ values with equal probability, the information in this random variable is $k$ bits, and from this we can deduce that $I(x) = - \log_2(p_x)$.  

---

## 1 answers

### Answer by username_2 on 2020-12-29. Score: 0 | Accepted answer

So definition of self-information is chosen to be as follows (shamelessly copied from wiki page of "Information content" ) :

1. An event with probability 100% is perfectly unsurprising and yields no information.
2. The less probable an event is, the more surprising it is and the more information it yields.
3. If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.

Point 3 of the definition answers your question.

If $A$ and $B$ independent events and C is an event where both $A$ and $B$ take place then
$$P(C) = P(A \cap B) = P(A)\cdot P(B)$$
On the other hand information from $C$ is defined as follows:
$$I(C) = I(A)+I(B)$$
Let $f(\cdot)$ be the functional form of the information expression, formally $$I(E) = f(P(E))$$
Therefore,
$$f(P(C)) = f(P(A)) + f(P(B))$$
$$ = f(P(A).P(B))$$

Can you find an $f(x)$ which is of the form $k\cdot x$, such that the above properties are satisfied ? I think the answer is no.

But I would like to add that I don't recall very neat reasons for why it has to be only $k\log(x)$ as well. i.e why only $f(x) = k\log x $ satisfies $f(x\cdot y) = f(x)+f(y)$

> **Comments:**
> **username_2**: For why its only Logx   --> https://www.reddit.com/r/math/comments/2x7an2/other_solutions_for_fxy_fx_fy/  

---

