<p>Your question is really about the relationship between <em>entropy</em> and <em>capacity</em>. Let's define the latter as the total number of possible states. The former is defined, on any finite distribution $P=(p_1,p_2,\ldots,p_n)$, by
$$ H(P) = \sum_{i=1}^n p_i\log(1/p_i).$$
When the distribution is <em>uniform</em> (i.e., $p_i\equiv1/n$), $H(P)=\log(n)$; this is the maximal value of $H(P)$ over all distributions $P$ on $n$ points (states). This is exactly the connection you ask about: in the case of the uniform distribution, the entropy equals the logarithm of the number of states. It can never be larger, but can be arbitrarily smaller for highly concentrated distributions.</p>

CC BY-SA 3.0
2017-02-08T14:02:23.167Z
2017-02-08T14:02:23.167Z
Aryeh
<p>I found this definition of Entropy:</p>

<blockquote>
  <p>Entropy measures the amount of random variation in a population. This
  means it measures the number of different states possible and the
  probability distribution across these states.</p>
</blockquote>

<p>So does that mean the more DOF are available, the higher the entropy is? There are 244 DOF in iris patterns with an entropy of about 3.2bits/mm². If there were more DOF, would the entropy rise?</p>

CC BY-SA 3.0
2017-02-08T18:59:40.510Z
This is not a definition, it's just a couple of imprecise sentences. And this is not a research level question in TCS.
Sasho Nikolov
CC BY-SA 3.0
2017-02-08T19:31:12.187Z
Indeed. Is it general policy not to answer such questions?
Aryeh
CC BY-SA 3.0
2017-02-09T03:48:02.440Z
@Aryeh I am not sure, but that is my impression.
Sasho Nikolov
CC BY-SA 3.0
2017-02-09T09:48:37.360Z
So why aren't they closing the question?
Aryeh
<definitions>
Are DOF and Entropy directly related?
4ndro1d
# Are DOF and Entropy directly related?

(Asked by: username_0 on 2017-02-08)

I found this definition of Entropy:

> Entropy measures the amount of random variation in a population. This
> means it measures the number of different states possible and the
> probability distribution across these states.

So does that mean the more DOF are available, the higher the entropy is? There are 244 DOF in iris patterns with an entropy of about 3.2bits/mm². If there were more DOF, would the entropy rise?

**Tags:** definitions

> **Comments:**
> **username_1**: This is not a definition, it's just a couple of imprecise sentences. And this is not a research level question in TCS.  
> **username_2**: Indeed. Is it general policy not to answer such questions?  
> **username_1**: @username_2 I am not sure, but that is my impression.  
> **username_2**: So why aren't they closing the question?  

---

## 1 answers

### Answer by username_2 on 2017-02-08. Score: 1

Your question is really about the relationship between *entropy* and *capacity*. Let's define the latter as the total number of possible states. The former is defined, on any finite distribution $P=(p\_1,p\_2,\ldots,p\_n)$, by
$$ H(P) = \sum\_{i=1}^n p\_i\log(1/p\_i).$$
When the distribution is *uniform* (i.e., $p\_i\equiv1/n$), $H(P)=\log(n)$; this is the maximal value of $H(P)$ over all distributions $P$ on $n$ points (states). This is exactly the connection you ask about: in the case of the uniform distribution, the entropy equals the logarithm of the number of states. It can never be larger, but can be arbitrarily smaller for highly concentrated distributions.


---

