<p>Under appropriate smoothness conditions (satisfied by the Boltzmann distribution), the maximum likelihood is asymptotically normal. Meaning: the vector <span class="math-container">$\sqrt n(\theta-\hat\theta)$</span> will have approximately normal distribution, with mean <span class="math-container">$0$</span> and covariance matrix given by the inverse Fisher matrix, see Section 9 here:
<a href="http://www.stat.cmu.edu/~larry/=stat705/Lecture9.pdf" rel="nofollow noreferrer">http://www.stat.cmu.edu/~larry/=stat705/Lecture9.pdf</a></p>

CC BY-SA 4.0
2018-10-10T21:17:53.677Z
I initially wanted to compute the Fisher matrix at $\theta$ but this seems to be non-trivial in this case -- I'd be curious if anyone can do it in closed form.
Aryeh
CC BY-SA 4.0
2018-10-10T14:37:22.140Z
2018-10-10T14:37:22.140Z
Aryeh
<p>I am trying to think through the number of samples that I would need to estimate the parameters of a Boltzmann partition function to a desirable precision.</p>
<p>Suppose that there are <span class="math-container">$N$</span> possible states of the world, with the probability of state <span class="math-container">$i$</span> being observed equal to</p>
<p><span class="math-container">$$Pr(i | \theta) = \frac{e^{-\theta_i}}{ \sum_{i=1}^N e^{-\theta_i}}.$$</span></p>
<p>I don't know what the values of <span class="math-container">$\{\theta_i\}_{i=1}^N$</span> are, but I can sample independent observations from the set <span class="math-container">$\{1,\dots,N\}$</span> of possible states of the world, distributed according to the distribution <span class="math-container">$Pr(i)$</span> given above.</p>
<p>Let's say I draw <span class="math-container">$m$</span> observations <span class="math-container">$X_1,...,X_m \in \{1,...,N\}$</span>, and define  the maximum likelihood estimator
<span class="math-container">$$\widehat{\theta} = \arg \max_{\theta'}  \sum_{j=1}^m  \log Pr(X_j | \theta)$$</span></p>
<p>When <span class="math-container">$m \to \infty$</span>, then  <span class="math-container">$\widehat{\theta} = \theta$</span>. However, I'm not sure how many samples <span class="math-container">$m$</span> I would need to ensure that the estimate <span class="math-container">$\widehat{\theta}$</span> is probably approximately correct. That is, how large does <span class="math-container">$m$</span> need to be in terms of <span class="math-container">$\epsilon,N,\delta$</span> to ensure that   <span class="math-container">$Pr ( \| \widehat{\theta} - \theta \|_{\infty} &gt; \epsilon ) &lt; \delta$</span>?</p>

<machine-learning><sample-complexity><pac-learning>
Sample complexity for learning Boltzmann Distribution parameters
Asterix
# Sample complexity for learning Boltzmann Distribution parameters

(Asked by: username_0 on 2018-10-10)

I am trying to think through the number of samples that I would need to estimate the parameters of a Boltzmann partition function to a desirable precision.

Suppose that there are $N$ possible states of the world, with the probability of state $i$ being observed equal to

$$Pr(i | \theta) = \frac{e^{-\theta\_i}}{ \sum\_{i=1}^N e^{-\theta\_i}}.$$

I don't know what the values of $\{\theta\_i\}\_{i=1}^N$ are, but I can sample independent observations from the set $\{1,\dots,N\}$ of possible states of the world, distributed according to the distribution $Pr(i)$ given above.

Let's say I draw $m$ observations $X\_1,...,X\_m \in \{1,...,N\}$, and define the maximum likelihood estimator
$$\widehat{\theta} = \arg \max\_{\theta'} \sum\_{j=1}^m \log Pr(X\_j | \theta)$$

When $m \to \infty$, then $\widehat{\theta} = \theta$. However, I'm not sure how many samples $m$ I would need to ensure that the estimate $\widehat{\theta}$ is probably approximately correct. That is, how large does $m$ need to be in terms of $\epsilon,N,\delta$ to ensure that $Pr ( \| \widehat{\theta} - \theta \|\_{\infty} > \epsilon ) < \delta$?

**Tags:** machine-learning, sample-complexity, pac-learning


---

## 1 answers

### Answer by username_1 on 2018-10-10. Score: 2 | Accepted answer

Under appropriate smoothness conditions (satisfied by the Boltzmann distribution), the maximum likelihood is asymptotically normal. Meaning: the vector $\sqrt n(\theta-\hat\theta)$ will have approximately normal distribution, with mean $0$ and covariance matrix given by the inverse Fisher matrix, see Section 9 here:
<http://www.stat.cmu.edu/~larry/=stat705/Lecture9.pdf>

> **Comments:**
> **username_1**: I initially wanted to compute the Fisher matrix at $\theta$ but this seems to be non-trivial in this case -- I'd be curious if anyone can do it in closed form.  

---

