<p>These questions arose when I was reading <a href="https://www.aeaweb.org/content/file?id=13476" rel="nofollow noreferrer">Online Appendix D</a> for the paper <em>Missing Events in Event Studies: Identifying the Effects of Partially-Measured News Surprises</em> by R.S. Gurkaynak, B. Kisacikoglu and J.H. Wright.</p>
<p>From what I have learnt, the usual exercise of Expectation-maximization with kalman filter is an iterative repetition of</p>
<ol>
<li><p>E-step: based on the estimates of parameters from the last M-step, use kalman filter to update the estimates of the latent variable;</p>
</li>
<li><p>M-step: use the estimates of latent variable from the E-step to get new estimates of parameters.</p>
</li>
</ol>
<p>In the paper cited above, however, the authors seem to use EMKF algorithm  to estimate the parameters with <strong>one single M-step</strong>.
Specifically, the authors aim to uncover the latent variable <span class="math-container">$f_t$</span> in the equation <span class="math-container">$$y_t=\beta's_t+\gamma d_tf_t+\varepsilon_t,$$</span>
where <span class="math-container">$y_t$</span> is an <span class="math-container">$n\times1$</span> vector, <span class="math-container">$s_t$</span> is a <span class="math-container">$K\times1$</span> vector, which are observed without error, <span class="math-container">$d_t$</span> is a dummy that is <span class="math-container">$1$</span> if there is an event at time <span class="math-container">$t$</span> or <span class="math-container">$0$</span> otherwise, <span class="math-container">$f_t$</span> is an iid <span class="math-container">$N(0,\sigma_f^2)$</span> latent variable, <span class="math-container">$\beta$</span> is a <span class="math-container">$K\times n$</span> matrix, <span class="math-container">$\gamma$</span> is an <span class="math-container">$n\times1$</span> vector, and <span class="math-container">$\varepsilon_t$</span> is iid with mean zero and diagonal variance-covariance matrix of <span class="math-container">$\Sigma_\varepsilon.$</span></p>
<p>The log-likelihood function is as follows:
<span class="math-container">$$\begin{aligned}l(\theta)=-\frac{1}{2}\Sigma_{t=1}^{T}\mathbf{1}(d_{t}=1)\{(y_{t}-\beta^{\prime}s_{t})^{\prime}(\Sigma_{\varepsilon}+\gamma\gamma^{\prime})^{-1}(y_{t}-\beta^{\prime}s_{t})\\&amp;+\log(|\Sigma_{\varepsilon}+\gamma\gamma^{\prime}|)\}\\&amp;+\mathbf{1}(d_{t}=0)\{y_{t}^{\prime}\Sigma_{\varepsilon}^{-1}y_{t}+\log(|\Sigma_{\varepsilon}|)\}\end{aligned}.$$</span>
To identify the parameters, the authors solve the following system of equations, which is obtained by maximize the log-likelihood function:
<span class="math-container">$$\beta=E(\mathbf{1}(d_t=1)s_ts_t^{\prime})^{-1}E(\mathbf{1}(d_t=1)s_ty_t^{\prime}),$$</span>
<span class="math-container">$$\Sigma_{\varepsilon}=E(y_{t}y_{t}^{\prime}\mathbf{1}(d_{t}=0)),$$</span> <span class="math-container">$$\gamma\gamma^{\prime}=E((y_{t}-\beta^{\prime}s_{t})(y_{t}-\beta^{\prime}s_{t})^{\prime}\mathbf{1}(d_{t}=1))-\Sigma_{\varepsilon}.$$</span></p>
<p>But <span class="math-container">$f_t$</span> does not appear in the equations above. Doesn't this mean to solve for <span class="math-container">$\beta$</span>, <span class="math-container">$\gamma$</span> and <span class="math-container">$\Sigma_{\epsilon}$</span> one does not actually need the kalman filter? Can anyone explain this to me?</p>

CC BY-SA 4.0
2024-10-06T02:02:27.650Z
I'm not sure about what you're saying the connection is between the KF and EM. The only connection I've heard of ( that doesn't mean that yours is incorrect ) is Stoffer's famous paper where he used the EM algorithm to estimate the variances in the KF framework. It's possible that this is exactly what you're talking about above. Check out Stoffer's paper to see if that sheds any light. I'll look for it and hopefully send a link after this comment.
mark leeds
CC BY-SA 4.0
2024-10-06T02:10:37.923Z
It's Shumway and Stoffer, 1982 at this page, near the bottom. I hope it is what you were talking about. https://dsstoffer.github.io/
mark leeds
CC BY-SA 4.0
2024-10-06T02:25:18.903Z
It says in the appendix on page 10: "as discussed in the text, we uncover $f_{t}$ using the kalman filter ... ". So, you should try to get your hands on the text if that's possible. Also, if what I linked to is what the paper is referring to, then I would think that Shumway and Stoffer, 1982 should be in the references of the paper.
mark leeds
CC BY-SA 4.0
2024-10-06T04:29:15.097Z
since $f_t$ only occurs when $d_t=1$ then you can get consistent estimates of the parameters as they are showing. You can then use those estimates to plug in as guesses for the Kalman Filter to then obtain $f_t$. Its a quite elegant solution to a special case.
Andrew M
CC BY-SA 4.0
2024-10-06T06:47:58.517Z
@markleeds Thank you for your comments. What you linked to is exactly what I referred to as a 'usual exercise of Expectation-maximization with kalman filter' in the description of my question (sorry if I failed to describe the EMKF algorithm correctly), and it seems to be quite distinct from the case in this paper. Also, I have read the text before reading the Appendices; the text does not mention any technical details, and that is exactly the reason why I turn to the Appendix.
zyy
CC BY-SA 4.0
2024-10-06T06:56:28.073Z
@AndrewM I think I get your point, but I am not sure. So actually, the estimates of the parameters does not rely on the estimates of $f_t$; they are merely obtained by maximizing the log-likelihood function. After that, estimates of the parameters are used in the *one-step* estimation of $f_t$ using Kalman filter. There is no need for iterations. Could you check if my understanding above is correct?
zyy
CC BY-SA 4.0
2024-10-06T09:16:44.300Z
Hi zyy: Okay, I wasn't sure if what I pointed to was distinct from whats in your paper or the same thing. It sounds distinct in which case I don't know what they're doing with the EM part of the algorithm. Chances are, it's somehow viewing the latent variable as missing data and then "uncovering it" using the EM algorithm but that's a total guess.  Hopefully, the Andrew M comments are helpful. I don't have time to try and understand the paper.
mark leeds
CC BY-SA 4.0
2024-10-07T17:31:28.603Z
I think there is a need for iterations here but the ML estimates provide a really nice first estimate. For example, for the $\gamma$ parameter you are able to get the scale $\gamma\gamma'$ but not the direction of $\gamma$ from these estimates alone. So there are likely gains from an iterative process where you feed in these estimates to get $f_t$ and then estimate $\gamma$ from there.
Andrew M
CC BY-SA 4.0
2024-10-10T14:10:02.870Z
@AndrewM Do you mean plug the KF estimates $f_{t|t}$ into the original equation, and solve by OLS to get new estimations of $\{\beta, \gamma, \Sigma_{\epsilon}\}$? That  does make sense to me. Thank you very much!
zyy
CC BY-SA 4.0
2024-10-10T14:14:00.420Z
@AndrewM I still have a problem. As is given by the authors in the appendix, the updating equations of the Kalman filter specify that the estimate of $f_t$ is $f_{t|t} = \gamma' (\Sigma_{\varepsilon}+d_t \gamma\gamma^{\prime})^{-1}(y_{t}-\beta^{\prime}s_{t}) d_t$. So one has to recover an estimation of $\gamma$ from the ML estimates of $\gamma \gamma'$ first. But I cannot see why this always work out. I think additional constraints must be put on the ML estimates * of $\gamma \gamma'$, (e.g., * is of rank $1$) to guarantee that the equation $\gamma \gamma' =* $ is solvable.
zyy
<macroeconomics><econometrics><time-series><causality>
On a non-standard application of Kalman filter
zyy
# On a non-standard application of Kalman filter

(Asked by: username_0 on 2024-10-05)

These questions arose when I was reading [Online Appendix D](https://www.aeaweb.org/content/file?id=13476) for the paper *Missing Events in Event Studies: Identifying the Effects of Partially-Measured News Surprises* by R.S. Gurkaynak, B. Kisacikoglu and J.H. Wright.

From what I have learnt, the usual exercise of Expectation-maximization with kalman filter is an iterative repetition of

1. E-step: based on the estimates of parameters from the last M-step, use kalman filter to update the estimates of the latent variable;
2. M-step: use the estimates of latent variable from the E-step to get new estimates of parameters.

In the paper cited above, however, the authors seem to use EMKF algorithm to estimate the parameters with **one single M-step**.
Specifically, the authors aim to uncover the latent variable $f\_t$ in the equation $$y\_t=\beta's\_t+\gamma d\_tf\_t+\varepsilon\_t,$$
where $y\_t$ is an $n\times1$ vector, $s\_t$ is a $K\times1$ vector, which are observed without error, $d\_t$ is a dummy that is $1$ if there is an event at time $t$ or $0$ otherwise, $f\_t$ is an iid $N(0,\sigma\_f^2)$ latent variable, $\beta$ is a $K\times n$ matrix, $\gamma$ is an $n\times1$ vector, and $\varepsilon\_t$ is iid with mean zero and diagonal variance-covariance matrix of $\Sigma\_\varepsilon.$

The log-likelihood function is as follows:
$$\begin{aligned}l(\theta)=-\frac{1}{2}\Sigma\_{t=1}^{T}\mathbf{1}(d\_{t}=1)\{(y\_{t}-\beta^{\prime}s\_{t})^{\prime}(\Sigma\_{\varepsilon}+\gamma\gamma^{\prime})^{-1}(y\_{t}-\beta^{\prime}s\_{t})\\&+\log(|\Sigma\_{\varepsilon}+\gamma\gamma^{\prime}|)\}\\&+\mathbf{1}(d\_{t}=0)\{y\_{t}^{\prime}\Sigma\_{\varepsilon}^{-1}y\_{t}+\log(|\Sigma\_{\varepsilon}|)\}\end{aligned}.$$
To identify the parameters, the authors solve the following system of equations, which is obtained by maximize the log-likelihood function:
$$\beta=E(\mathbf{1}(d\_t=1)s\_ts\_t^{\prime})^{-1}E(\mathbf{1}(d\_t=1)s\_ty\_t^{\prime}),$$
$$\Sigma\_{\varepsilon}=E(y\_{t}y\_{t}^{\prime}\mathbf{1}(d\_{t}=0)),$$ $$\gamma\gamma^{\prime}=E((y\_{t}-\beta^{\prime}s\_{t})(y\_{t}-\beta^{\prime}s\_{t})^{\prime}\mathbf{1}(d\_{t}=1))-\Sigma\_{\varepsilon}.$$

But $f\_t$ does not appear in the equations above. Doesn't this mean to solve for $\beta$, $\gamma$ and $\Sigma\_{\epsilon}$ one does not actually need the kalman filter? Can anyone explain this to me?

**Tags:** macroeconomics, econometrics, time-series, causality

> **Comments:**
> **username_1**: I'm not sure about what you're saying the connection is between the KF and EM. The only connection I've heard of ( that doesn't mean that yours is incorrect ) is Stoffer's famous paper where he used the EM algorithm to estimate the variances in the KF framework. It's possible that this is exactly what you're talking about above. Check out Stoffer's paper to see if that sheds any light. I'll look for it and hopefully send a link after this comment.  
> **username_1**: It's Shumway and Stoffer, 1982 at this page, near the bottom. I hope it is what you were talking about. https://dsstoffer.github.io/  
> **username_1**: It says in the appendix on page 10: "as discussed in the text, we uncover $f_{t}$ using the kalman filter ... ". So, you should try to get your hands on the text if that's possible. Also, if what I linked to is what the paper is referring to, then I would think that Shumway and Stoffer, 1982 should be in the references of the paper.  
> **username_2**: since $f_t$ only occurs when $d_t=1$ then you can get consistent estimates of the parameters as they are showing. You can then use those estimates to plug in as guesses for the Kalman Filter to then obtain $f_t$. Its a quite elegant solution to a special case.  
> **username_0**: @markleeds Thank you for your comments. What you linked to is exactly what I referred to as a 'usual exercise of Expectation-maximization with kalman filter' in the description of my question (sorry if I failed to describe the EMKF algorithm correctly), and it seems to be quite distinct from the case in this paper. Also, I have read the text before reading the Appendices; the text does not mention any technical details, and that is exactly the reason why I turn to the Appendix.  
> **username_0**: @AndrewM I think I get your point, but I am not sure. So actually, the estimates of the parameters does not rely on the estimates of $f_t$; they are merely obtained by maximizing the log-likelihood function. After that, estimates of the parameters are used in the *one-step* estimation of $f_t$ using Kalman filter. There is no need for iterations. Could you check if my understanding above is correct?  
> **username_1**: Hi username_0: Okay, I wasn't sure if what I pointed to was distinct from whats in your paper or the same thing. It sounds distinct in which case I don't know what they're doing with the EM part of the algorithm. Chances are, it's somehow viewing the latent variable as missing data and then "uncovering it" using the EM algorithm but that's a total guess.  Hopefully, the username_2 comments are helpful. I don't have time to try and understand the paper.  
> **username_2**: I think there is a need for iterations here but the ML estimates provide a really nice first estimate. For example, for the $\gamma$ parameter you are able to get the scale $\gamma\gamma'$ but not the direction of $\gamma$ from these estimates alone. So there are likely gains from an iterative process where you feed in these estimates to get $f_t$ and then estimate $\gamma$ from there.  
> **username_0**: @AndrewM Do you mean plug the KF estimates $f_{t|t}$ into the original equation, and solve by OLS to get new estimations of $\{\beta, \gamma, \Sigma_{\epsilon}\}$? That  does make sense to me. Thank you very much!  
> **username_0**: @AndrewM I still have a problem. As is given by the authors in the appendix, the updating equations of the Kalman filter specify that the estimate of $f_t$ is $f_{t|t} = \gamma' (\Sigma_{\varepsilon}+d_t \gamma\gamma^{\prime})^{-1}(y_{t}-\beta^{\prime}s_{t}) d_t$. So one has to recover an estimation of $\gamma$ from the ML estimates of $\gamma \gamma'$ first. But I cannot see why this always work out. I think additional constraints must be put on the ML estimates * of $\gamma \gamma'$, (e.g., * is of rank $1$) to guarantee that the equation $\gamma \gamma' =* $ is solvable.  

---

## 0 answers

