<p>Suppose <span class="math-container">$x_i \sim f(\theta,\nu_i)$</span> and <span class="math-container">$y_i \sim g(\tau,\theta,\nu_i)$</span> and <span class="math-container">$\nu_i \sim N(0,\sigma)$</span>. Importantly, <span class="math-container">$f$</span> is smooth in <span class="math-container">$\theta$</span>, but <span class="math-container">$g$</span> is not.</p>
<p>We have data on <span class="math-container">$x_i$</span>, <span class="math-container">$y_i$</span> and the goal is to estimate <span class="math-container">$A=(\tau,\theta,\sigma)$</span>.</p>
<p>The marginal likelihoods are:
<span class="math-container">$$L_{x_i}(\theta,\sigma) = \int f(\theta,\nu_i) \ d\Phi(\nu_{i};0,\sigma)$$</span>
<span class="math-container">$$L_{y_i}(\tau,\theta,\sigma) = \int g(\tau,\theta,\nu_i) \ d\Phi(\nu_{i};0,\sigma)$$</span></p>
<p>However, maximizing the individual likelihoods would lead to biased estimates of <span class="math-container">$\sigma$</span> due to the correlation via <span class="math-container">$\nu_i$</span> (Is this correct?). Alternatively, the joint likelihood is then:
<span class="math-container">$$L(\tau,\theta,\sigma) = \prod_i \int f(\theta,\nu_i) g(\tau,\theta,\nu_i) \ d\Phi(\nu_{i};0,\sigma)$$</span></p>
<p>To maximize this (simulated) likelihood, given <span class="math-container">$R$</span> draws of <span class="math-container">$\nu_i$</span>, the EM algorithm develops a surrogate function:
<span class="math-container">$$Q(A | A^m) =  \sum_{i,r} w_{i,r}^m \ln f(\theta,\nu_{ir}) g(\tau,\theta,\nu_{ir}) + \sum_{i,r} w_{i,r}^m \ln \phi(\nu_{ir};0,\sigma)$$</span>
where
<span class="math-container">$$w_{ir}^m = \frac{f(\theta^m,\nu_{ir}) g(\tau^m,\theta^m,\nu_{ir})}{\sum_{r'}f(\theta^m,\nu_{ir'}) g(\tau^m,\theta^m,\nu_{ir'}) } $$</span>
Iteratively maximizing this function with respect to parameters, converges to the maximum of the simulated joint likelihood.</p>
<p>Call the first sum <span class="math-container">$Q_1(\theta,\tau; A^m)$</span> and the second <span class="math-container">$Q_2(\sigma; A^m)$</span>, Maximizing <span class="math-container">$Q_2$</span> is trivial, and reduces to the (weighted) sample standard deviation of <span class="math-container">$\nu_{ir}$</span>.</p>
<p>Maximizing <span class="math-container">$Q_1$</span> over <span class="math-container">$\tau,\theta$</span> is challenging because <span class="math-container">$g$</span> is not smooth in <span class="math-container">$\theta$</span>. Instead, we decompose this into <span class="math-container">$\tilde{Q}_1$</span>:
<span class="math-container">$$\tilde{Q}_1(\theta,\tau; A^m) = \sum_{i,r} w_{i,r}^m \ln f(\theta,\nu_{ir}) + \sum_{i,r} w_{i,r}^m \ln g(\tau,\theta^m,\nu_{ir})$$</span>
where the <span class="math-container">$\theta$</span> in the second sum is replaced by <span class="math-container">$\theta^m$</span>. Now, the first sum can be maximized because <span class="math-container">$f$</span> is smooth in <span class="math-container">$\theta$</span> and the second sum can be maximized because <span class="math-container">$g$</span> is smooth in <span class="math-container">$\tau$</span> (and no longer depends on <span class="math-container">$\theta$</span>).</p>
<p>The intuition is that, because <span class="math-container">$\tau,\theta$</span> are fixed parameters, the data <span class="math-container">$x_i$</span> provides information to separately identify <span class="math-container">$\theta$</span> and <span class="math-container">$y_i$</span> provides information to separately identify <span class="math-container">$\tau$</span> conditional on <span class="math-container">$\nu_i$</span>. Meanwhile, the estimation of <span class="math-container">$\sigma$</span>, which comes from maximizing <span class="math-container">$Q_2$</span>, still accounts for the joint likelihood via <span class="math-container">$w_{ir}^m$</span>.</p>
<p>I understand that this would be a loss in efficiency, as we are essentially dropping data by not using <span class="math-container">$y_i$</span> to estimate <span class="math-container">$\theta$</span>. However, it seems reasonable that the estimate would still be unbiased and consistent. I'm confused about how to argue this or if I'm missing something completely.</p>

CC BY-SA 4.0
2023-09-19T07:19:09.943Z
Clarification question: as $\nu$ has an $i$ subscript, it looks like that you have an incidental parameter problem here. Why do you also use notation $\nu_{ir}$? What is $\Phi(0,\sigma)$ ? It depends on two parameters only? why is it evaluated at $0$?
Bertrand
CC BY-SA 4.0
2023-09-19T16:47:06.963Z
$\nu_i$ is an unobserved random effect, with known mean zero and unknown variance $\sigma$. Its CDF is $\Phi(\nu_i; 0,\sigma)$. The $r$ subscript denotes the $r$th draw of $\nu_i$ in the EM algorithm (which takes $R$ random draws of $\nu_i$ in each iteration). So it should only appear when I define $Q$, thanks for the edit!
highly-nontrivial
CC BY-SA 4.0
2023-09-21T07:46:08.303Z
What is $A^m$ ? Where is $\theta^m$ in the expression of $\tilde{Q}_1$? If the dimension of $\theta$ is small, I would tackle the optimization problem using Nelder-Mead or other numerical methods which do not rely on derivatives.
Bertrand
<econometrics>
Using an EM algorithm for a random parameters model with mixed data
highly-nontrivial
# Using an EM algorithm for a random parameters model with mixed data

(Asked by: username_0 on 2023-09-18)

Suppose $x\_i \sim f(\theta,\nu\_i)$ and $y\_i \sim g(\tau,\theta,\nu\_i)$ and $\nu\_i \sim N(0,\sigma)$. Importantly, $f$ is smooth in $\theta$, but $g$ is not.

We have data on $x\_i$, $y\_i$ and the goal is to estimate $A=(\tau,\theta,\sigma)$.

The marginal likelihoods are:
$$L\_{x\_i}(\theta,\sigma) = \int f(\theta,\nu\_i) \ d\Phi(\nu\_{i};0,\sigma)$$
$$L\_{y\_i}(\tau,\theta,\sigma) = \int g(\tau,\theta,\nu\_i) \ d\Phi(\nu\_{i};0,\sigma)$$

However, maximizing the individual likelihoods would lead to biased estimates of $\sigma$ due to the correlation via $\nu\_i$ (Is this correct?). Alternatively, the joint likelihood is then:
$$L(\tau,\theta,\sigma) = \prod\_i \int f(\theta,\nu\_i) g(\tau,\theta,\nu\_i) \ d\Phi(\nu\_{i};0,\sigma)$$

To maximize this (simulated) likelihood, given $R$ draws of $\nu\_i$, the EM algorithm develops a surrogate function:
$$Q(A | A^m) = \sum\_{i,r} w\_{i,r}^m \ln f(\theta,\nu\_{ir}) g(\tau,\theta,\nu\_{ir}) + \sum\_{i,r} w\_{i,r}^m \ln \phi(\nu\_{ir};0,\sigma)$$
where
$$w\_{ir}^m = \frac{f(\theta^m,\nu\_{ir}) g(\tau^m,\theta^m,\nu\_{ir})}{\sum\_{r'}f(\theta^m,\nu\_{ir'}) g(\tau^m,\theta^m,\nu\_{ir'}) } $$
Iteratively maximizing this function with respect to parameters, converges to the maximum of the simulated joint likelihood.

Call the first sum $Q\_1(\theta,\tau; A^m)$ and the second $Q\_2(\sigma; A^m)$, Maximizing $Q\_2$ is trivial, and reduces to the (weighted) sample standard deviation of $\nu\_{ir}$.

Maximizing $Q\_1$ over $\tau,\theta$ is challenging because $g$ is not smooth in $\theta$. Instead, we decompose this into $\tilde{Q}\_1$:
$$\tilde{Q}\_1(\theta,\tau; A^m) = \sum\_{i,r} w\_{i,r}^m \ln f(\theta,\nu\_{ir}) + \sum\_{i,r} w\_{i,r}^m \ln g(\tau,\theta^m,\nu\_{ir})$$
where the $\theta$ in the second sum is replaced by $\theta^m$. Now, the first sum can be maximized because $f$ is smooth in $\theta$ and the second sum can be maximized because $g$ is smooth in $\tau$ (and no longer depends on $\theta$).

The intuition is that, because $\tau,\theta$ are fixed parameters, the data $x\_i$ provides information to separately identify $\theta$ and $y\_i$ provides information to separately identify $\tau$ conditional on $\nu\_i$. Meanwhile, the estimation of $\sigma$, which comes from maximizing $Q\_2$, still accounts for the joint likelihood via $w\_{ir}^m$.

I understand that this would be a loss in efficiency, as we are essentially dropping data by not using $y\_i$ to estimate $\theta$. However, it seems reasonable that the estimate would still be unbiased and consistent. I'm confused about how to argue this or if I'm missing something completely.

**Tags:** econometrics

> **Comments:**
> **username_1**: Clarification question: as $\nu$ has an $i$ subscript, it looks like that you have an incidental parameter problem here. Why do you also use notation $\nu_{ir}$? What is $\Phi(0,\sigma)$ ? It depends on two parameters only? why is it evaluated at $0$?  
> **username_0**: $\nu_i$ is an unobserved random effect, with known mean zero and unknown variance $\sigma$. Its CDF is $\Phi(\nu_i; 0,\sigma)$. The $r$ subscript denotes the $r$th draw of $\nu_i$ in the EM algorithm (which takes $R$ random draws of $\nu_i$ in each iteration). So it should only appear when I define $Q$, thanks for the edit!  
> **username_1**: What is $A^m$ ? Where is $\theta^m$ in the expression of $\tilde{Q}_1$? If the dimension of $\theta$ is small, I would tackle the optimization problem using Nelder-Mead or other numerical methods which do not rely on derivatives.  

---

## 0 answers

