<p>I am writing a program using high-dimensional volume (HDV) estimator to estimate entropy and mutual information for variable selection. Let $ D = (x^i_1, x^i_2, ..., x^i_M)$, N is the number of data points in the $M$ dimensional joint space.</p>

<p>The HDV estimators based on the equation as below:</p>

<p>\begin{aligned}
H(X) ≈ −ψ(k) + ψ(N) + \frac{1}{N}{\sum_{i=1}^{N}{logB_m(r^{i}_{x^m})}^m}
\end{aligned}</p>

<p>$X$ is the random variable, $H(X)$ is the entropy of $X$, $ψ(.)$ is the digamma function,  $k=1$ for HDV estimator, $N$ is the number of observations, $B_m$ is the volume of m-dimensional unit ball $r^{i}_{x^m}$ is the minimum distance in each dimension:</p>

<p>\begin{aligned}
r^{i}_{x^m} = min(||x^i_m-x^{'i}_m||)
\end{aligned}</p>

<p>where $x^i_m \neq x^{&#39;i}_m$ and $x^i_m , x^{&#39;i}_m \in D$</p>

<p>My question is that when it is applied to estimate the entropy of Gaussian distribution with mean = 0 and standard deviation = 1, the theoretical result of entropy is around 1.4189. I cannot get the result because $−ψ(1)≈ 0.5772156$ (Euler-Mascheroni constant), $ψ(N=1000)≈ 6.9072$ (Say 1,000 of data points) and I think that $\frac{1}{N}{\sum_{i=1}^{N}{logB_m(r^{i}_{x^m})}^m}$ cannot be negative as it is about distance and volume. What is wrong of my understanding?</p>

CC BY-SA 3.0
2011-08-31T16:31:29.150Z
might be more suitable for the stats.SE.
Kaveh
CC BY-SA 3.0
2011-08-31T16:57:37.177Z
I agree. you might try posting it there.
Suresh Venkat
CC BY-SA 3.0
2011-08-31T20:28:40.797Z
I cannot comment on the estimator definition, but note that the volume of an $m$-dimensional unit ball is actually exponentially decreasing in $m$. I think $\log B_m$ is on the order of $-O(m\log m)$.
Sasho Nikolov
<it.information-theory><st.statistics><na.numerical-analysis>
High Dimensional Volume (HDV) estimator for Entropy estimation
JYJ
# High Dimensional Volume (HDV) estimator for Entropy estimation

(Asked by: username_0 on 2011-08-31)

I am writing a program using high-dimensional volume (HDV) estimator to estimate entropy and mutual information for variable selection. Let $ D = (x^i\_1, x^i\_2, ..., x^i\_M)$, N is the number of data points in the $M$ dimensional joint space.

The HDV estimators based on the equation as below:

\begin{aligned}
H(X) ≈ −ψ(k) + ψ(N) + \frac{1}{N}{\sum\_{i=1}^{N}{logB\_m(r^{i}\_{x^m})}^m}
\end{aligned}

$X$ is the random variable, $H(X)$ is the entropy of $X$, $ψ(.)$ is the digamma function, $k=1$ for HDV estimator, $N$ is the number of observations, $B\_m$ is the volume of m-dimensional unit ball $r^{i}\_{x^m}$ is the minimum distance in each dimension:

\begin{aligned}
r^{i}\_{x^m} = min(||x^i\_m-x^{'i}\_m||)
\end{aligned}

where $x^i\_m \neq x^{'i}\_m$ and $x^i\_m , x^{'i}\_m \in D$

My question is that when it is applied to estimate the entropy of Gaussian distribution with mean = 0 and standard deviation = 1, the theoretical result of entropy is around 1.4189. I cannot get the result because $−ψ(1)≈ 0.5772156$ (Euler-Mascheroni constant), $ψ(N=1000)≈ 6.9072$ (Say 1,000 of data points) and I think that $\frac{1}{N}{\sum\_{i=1}^{N}{logB\_m(r^{i}\_{x^m})}^m}$ cannot be negative as it is about distance and volume. What is wrong of my understanding?

**Tags:** it.information-theory, st.statistics, na.numerical-analysis
> **Comments:**
> **username_1**: might be more suitable for the stats.SE.  
> **username_2**: I agree. you might try posting it there.  
> **username_3**: I cannot comment on the estimator definition, but note that the volume of an $m$-dimensional unit ball is actually exponentially decreasing in $m$. I think $\log B_m$ is on the order of $-O(m\log m)$.  

---

## 0 answers

