<p>You're confusing the Shannon entropy of a discrete probability distribution with the differential entropy of a continuous probability distribution. The minimum distribution length is only given by the Shannon entropy for discrete probability distributions.</p>

<p>What the fact that the <strong><em>differential entropy</em></strong> is 0 for this probability distribution means is that for large $k$, you can approximate this random variable with an expected error of $1/2^k$ using $k$ bits. If the differential entropy were $\ell$, you could approximate it with an error of $1/2^k$ using $k+\ell$ bits.</p>

CC BY-SA 3.0
2014-05-31T04:09:44.300Z
Thank you! It's clear that I can't transmit continuously distributed messages without error, so yes, my terminology is inaccurate, apologies. My bigger concern is how to represent mapping $\mathbb{R} \mapsto [0,1]$ (ie full $F(x)$) and how to bound its entropy.
Preston
CC BY-SA 3.0
2014-05-31T12:12:29.910Z
I don't understand your concern. You're mapping the entire real line to the unit interval? How? Do maps have entropies associated with them? I thought probability distributions had entropies associated with them.
Peter Shor 
CC BY-SA 3.0
2014-05-31T13:30:58.023Z
Say, my friend asks me about the distance to the nearest gas station in meters. I know that it's represented by a certain probability distribution, e.g. Laplace. I would like to pass this information to my friend without giving the formula of the corresponding CDF (my friend hates math). The entropy we are interested in is the entropy of this message to my friend.
Preston
CC BY-SA 3.0
2014-05-31T03:40:19.123Z
2014-05-31T03:40:19.123Z
Peter Shor 
<p>The minimum description length of the realization of a random variable $x$ is given by the entropy of its probability mass function. It can be asymptotically communicated at this rate with an optimal coding. But that's for just one $x$ from the optimal coding table. How do we cover the whole support efficiently? (Of course, we'll have issues with quantization of real valued $x$-s in our code table, however optimized it's going to be.)</p>

<p>Viewing it differently, entropy of the representation of a generic probability distribution with CDF $F(x)$, where $F(x) \sim p(F(x))$ on some space of distributions $D$, seems to be bounded from above by entropy of uniform $p$ on space $D$: $-\int_D{p(F(x))\ln{p(F(x))}dF(x)}$. But this is clearly not a sharp maximum description length bound (probably need to introduce some reasonable restrictions).</p>

<p>Looks like I'm missing something obvious, any ideas would be appreciated...</p>

<p>Oh, and this has nothing to do with differential vs discrete entropy, the same question can be posed for the latter case.</p>

<p>[Updated after Peter Shor's comments; also cleaned up language and formulations a bit.]</p>

<reference-request><it.information-theory>
description of continuous probability distribution
Preston
# description of continuous probability distribution

(Asked by: username_0 on 2014-05-31)

The minimum description length of the realization of a random variable $x$ is given by the entropy of its probability mass function. It can be asymptotically communicated at this rate with an optimal coding. But that's for just one $x$ from the optimal coding table. How do we cover the whole support efficiently? (Of course, we'll have issues with quantization of real valued $x$-s in our code table, however optimized it's going to be.)

Viewing it differently, entropy of the representation of a generic probability distribution with CDF $F(x)$, where $F(x) \sim p(F(x))$ on some space of distributions $D$, seems to be bounded from above by entropy of uniform $p$ on space $D$: $-\int\_D{p(F(x))\ln{p(F(x))}dF(x)}$. But this is clearly not a sharp maximum description length bound (probably need to introduce some reasonable restrictions).

Looks like I'm missing something obvious, any ideas would be appreciated...

Oh, and this has nothing to do with differential vs discrete entropy, the same question can be posed for the latter case.

[Updated after Peter Shor's comments; also cleaned up language and formulations a bit.]

**Tags:** reference-request, it.information-theory

---

## 1 answers

### Answer by username_1 on 2014-05-31. Score: 9

You're confusing the Shannon entropy of a discrete probability distribution with the differential entropy of a continuous probability distribution. The minimum distribution length is only given by the Shannon entropy for discrete probability distributions.

What the fact that the ***differential entropy*** is 0 for this probability distribution means is that for large $k$, you can approximate this random variable with an expected error of $1/2^k$ using $k$ bits. If the differential entropy were $\ell$, you could approximate it with an error of $1/2^k$ using $k+\ell$ bits.

> **Comments:**
> **username_0**: Thank you! It's clear that I can't transmit continuously distributed messages without error, so yes, my terminology is inaccurate, apologies. My bigger concern is how to represent mapping $\mathbb{R} \mapsto [0,1]$ (ie full $F(x)$) and how to bound its entropy.  
> **username_1**: I don't understand your concern. You're mapping the entire real line to the unit interval? How? Do maps have entropies associated with them? I thought probability distributions had entropies associated with them.  
> **username_0**: Say, my friend asks me about the distance to the nearest gas station in meters. I know that it's represented by a certain probability distribution, e.g. Laplace. I would like to pass this information to my friend without giving the formula of the corresponding CDF (my friend hates math). The entropy we are interested in is the entropy of this message to my friend.  

---

