<p>I do not think that macroeconomic analysis would be 'less trust worthy'. In the same ways as you can solve the issue of omitted variables and reverse causality in microeconomics you can do it macroeconomics as well. Macroeconomics, is not just vector autoregressions (VAR - which by the way corrects for endogeneity). Simultaneous equation techniques, even difference-in-difference (DiD) and regression discontinuity (RD) etc are used even in macro literature even if they are less common (see this recent paper on identification in macroeconomics Nakamura &amp; Steinsson, 2018).</p>
<p>In addition I think you are actually exaggerating quality of quasi-experimental methods such as IV or DiD. For example, this recent paper shows that large portion of IV papers uses weak instruments as the rule of thumb that <span class="math-container">$F$</span>-stat. should be above 10 is invalid and  true threshold is around <span class="math-container">$F&gt;105$</span> (see here <a href="https://arxiv.org/abs/2010.05058" rel="nofollow noreferrer">Lee McCrary Moreira &amp; Porter 2020</a>). In addintion DiD are notorious for <span class="math-container">$p$</span>-hacking and other issues,  <a href="https://academic.oup.com/qje/article/119/1/249/1876068" rel="nofollow noreferrer">Duflo &amp; Mullainathan (2004)</a> literally have paper titled: &quot;How much should we trust differences-in-differences estimates?&quot; Spoiler alert their answer is not as much as people usually do.</p>
<p>I think it would be fair to say that a reasonable person could defend saying that typical VAR has probably on average less issues that let's say DiD (but to be clear I am not claiming that as  even this lacks nuance as the model selection and specification is always case dependent there is no one model uber ales - even RCTs have a problem when it comes to generalization of results).</p>
<p>When it comes to just empirics in general I could not find any paper that would show that empirical research in macroeconomics is less replicable than in microeconomics. I would also take an issue with claim that macroeconomic models cannot control for all relevant variables. There are macro models (e.g. dynamic stochastic general equilibrium  (DSGE)) that include scores of different equations and even more variables. Next, even despite of this when it comes to omitted variable bias you might have a valid point although a lot of attention is being paid to the issue it might sometimes be hard to solve, but with the reverse causality your accusation that macroeconomics does not pay enough attention is unfounded. Most published research in macroeconomics pays very large attention to reverse causality (especially in present day, maybe 50-60 years ago that would be valid point as well). In fact that is why models and estimation techniques like VAR, DSGE, IV, GMM and many other simultaneous equations approaches are so popular and dominate macroeconomics, so this second charge is simply unfounded.</p>
<p>However, when it comes to forecasts I would agree that macroeconomic forecasting is for sure less accurate then microeconomic forecasting (e.g. forecasting aggregate output vs firm forecasting demand for its onw products). However, there the fault would not be in the models themselves necessary but rather with the fact that macroeconomic aggregates contain more noise, are harder to measure, and are reported on lower frequency and of course with less data it is harder to build good forecasting model.</p>
<p>Lastly, it is not really easy to qualify how much you should trust a model in any field. You should approach any model and set of results with healthy doze of skepticism, be they from VAR or RTC or any other approach. Different situations will require different identification strategies. Furthermore, you should not put all faith into single paper but wait for replications/put the paper into larger context of pre-existing literature. I also tried to find some work that would compare replicability of microeconomics vs macroeconomics, but unfortunately there does not seem to be such paper but I would not expect macroeconomics having much worse replicability rates then microeconomics per se.</p>

CC BY-SA 4.0
2020-11-04T15:17:09.773Z
I love reading your answers and comments because you know more about econometrics than I'll ever know and they're chock full of wisdom.  Then again, that's not a huge compliment because I know very little :) . But why is your focus on reproducibility ? Something can be reproduced and still not be high quality. (i.e., bogus or useless ). Shouldn't the focus, in the end, be forecasting ability ?
mark leeds
CC BY-SA 4.0
2020-11-04T15:22:17.957Z
@markleeds thanks for the compliments, you make me blush :). Now to respond to the comment: when we talk about reproducibility we do not mean just take the same data use the same method and get the same result - of course this can be always done even with p-hacked papers. Reproducibility of a result is often colloquially used even for a situation where someone tries to test the same relationship even with slightly different method the results should be similar, or trying to test the same relationship on different sample and see if results are reasonably robust.
1muflon1
CC BY-SA 4.0
2020-11-04T15:27:28.430Z
@markleeds for example colloquially people often refer to Nuemark and Wascher study that tried to reproduce the minimum wage study of Card and Kruger, using completely different dataset as reproduction study - card and Kruger used self reported data while nuemark and wascher used hard payroll data. If two studies like this agree the result is considered to be reproducible and if not it would not be (at least colloquially in economics - I am not sure if maybe wider statistical literature on uses reproducibility also in more narrow sense.
1muflon1
CC BY-SA 4.0
2020-11-04T15:33:47.203Z
Maybe I did not express this well, I am in no way putting micro over macro theory in this question. Also I know that one can use quasi experimental methods for macroeconomic issues. I am only talking about time series econometrics vs. the quasi experimental stuff. I also know the DiD paper, but I think since 2004 a lot has gotten better (of course nothing is ever good enough though). Lastly, I think the advantage of the quasi exp. methods is that weaknesses can be revealed quite easily, e.g. a good paper will show and discuss how weak/strong an instrument is.
ArOk
CC BY-SA 4.0
2020-11-04T15:44:00.733Z
@ArOk but good papers will also show how good/bad VAR or error correction model or DSGE is. If you are asking if it is any harder to check whether the models are applied correctly in macro or micro then answer is resounding no. However, I kinda can see why you might have this perception. In order to learn time series analysis you first need to learn cross-sectional analysis so you most likely been exposed to content that covered cross-sectional analysis in great detail but only grazed time series models - which will then be covered in separate classes.
1muflon1
CC BY-SA 4.0
2020-11-04T22:08:24.427Z
@1muflon1: I got it now. In statistics field, when one refs to reproducibility, they are strictly talking about getting the same EXACT results. Thanks for clarification. Note though that I would argue that your definition of reproducibility is questionable because two papers that reproduce each others results by your definition could still have their own seperate issues. ( questionable assumptions,  whatever ).
mark leeds
CC BY-SA 4.0
2020-11-04T22:16:28.527Z
@markleeds is there in statistics maybe better word for it? I would not mind to also edit that in to make it more clear. And I also agree with you but the idea is that if different approaches with all their flaws/issues manage to get the same/similar results there will likely be some effect there. I mean each result can be sort of think of as an observation like in a meta study. Yes of course there are still issues like publication bias but I would still argue seeing multiple similar results increases the confidence that we are really measuring some true effect
1muflon1
CC BY-SA 4.0
2020-11-05T16:48:56.653Z
@1muflon1: good point. I basically agree that the more evidence the stronger the evidence. Unfortunately, I don't know what the word is in statistics ? In fact, I'm not sure if there is one word but atleast I understand what your definition of reproducibility is. Unfortunately ( or fortunately ), due to R and github,  reproducibility in statistical academics is getting much hype these days but it has the "duplicate the results" meaning that I mentioned earlier. I'll try to look around when I have some time but there may not be one ? All the best and I'm sorry that I can't help there.
mark leeds
CC BY-SA 4.0
2020-11-18T11:01:34.217Z
@markleeds, what you describe might be the programming rather than statistical understanding of *reproducibility*.
Richard Hardy
CC BY-SA 4.0
2020-11-19T12:24:22.653Z
Hi Richard: In statistical world, the term means EXACTLY DUPLICATING the statistical results of someone else, given the same data. So, yes, definitely programming ( R is kind of standard in stat world )  is the key to it if that's what you meant.
mark leeds
CC BY-SA 4.0
2020-11-19T12:36:46.967Z
@RichardHardy I am tagging you since mark forgot to. If it would not be bothersome could you respond? I would be interested in getting to the bottom of this semantic dispute
1muflon1
CC BY-SA 4.0
2020-11-19T12:40:59.240Z
@markleeds I found this answer on cross-validated it says reproducibility can have both meanings - one is considered to be narrow meaning and one is considered to be broad meaning https://stats.stackexchange.com/a/15006/89502 is it possible that you and Richard just focus on one of these different views
1muflon1
CC BY-SA 4.0
2020-11-20T13:53:47.443Z
@1muflon1: I didn't know that there were two views ( on the meaning in statistics anyway. I can see how there would be two views in general ) so my apologies. Maybe your view was in statistics in previous years and then, say in the last decade or so, the one I was talking about came around, so they're both around.   I'll check out the link. My mistake and sorry to lead you astray if I did.
mark leeds
CC BY-SA 4.0
2020-11-20T14:04:34.467Z
@1muflon1: I read the link and using reproducible "analysis" or "data analysis" etc does make sense. It would make the meaning clearer for sure.
mark leeds
CC BY-SA 4.0
2020-11-04T15:08:41.050Z
2020-11-18T11:01:45.227Z
2020-11-18T11:01:45.227Z
1muflon1
<p>I am a student of economics in my masters and I have learned quite a lot about microeconometrics (I mean mainly quasi-experimental methods / causality determination). Here my current understanding is that obviously you can't ever fully trust any analysis, but basically you can get to a point where results are relatively convincing, i.e. when one can make a strong argument for the identification assumption to hold (so parallel trends in a Dif-in-Dif, exclusion restriction in an IV approach, etc.). Alternatively, you get identification through a well designed RCT. So basically you can find ways to solve problems of reverse causality and omitted variables.</p>
<p>I have now recently started to learn more about macroeconometrics (i.e. time series methods) and it kind of appears to me that reverse causality and omitted variable bias kind of get ignored here (I am thinking of 3-4 variable VARs that obvisouly don't capture all the variables that might explain e.g. output or inflation). Is there some mathematical argument that makes these problems disappear magically? Or is macroeconometrics in general less convincing?</p>
<p>I have heard of the credibility revolution in empirical micro and that nothing similar happened in macro, but how much can we trust results from something like an IRF estimation or how convincing are arguments based on IRF results?</p>
<p>PS. I know that in forecasting you can be kind of agnostic to any causality issues, so in this question  I am not asking about forecasts. Also I am not putting microeconomics vs macroeconomics in this question. I am only talking about two different econometric approaches.</p>

CC BY-SA 4.0
2020-11-18T12:09:52.737Z
Most time series analysis depends upon an assumption that statistical properties of tomorrow are like the statistical properties of today, regardless of how parsimonious or how rich your model is in terms of parameters. That doesn't make me a sceptic on econometrics, but instead, it is important to understand it's limitations.
EB3112
<econometrics><time-series><empirical-evidence><empirical-methods>
How much can we trust macroeconometric analysis?
ArOk
# How much can we trust macroeconometric analysis?

(Asked by: username_0 on 2020-11-04)

I am a student of economics in my masters and I have learned quite a lot about microeconometrics (I mean mainly quasi-experimental methods / causality determination). Here my current understanding is that obviously you can't ever fully trust any analysis, but basically you can get to a point where results are relatively convincing, i.e. when one can make a strong argument for the identification assumption to hold (so parallel trends in a Dif-in-Dif, exclusion restriction in an IV approach, etc.). Alternatively, you get identification through a well designed RCT. So basically you can find ways to solve problems of reverse causality and omitted variables.

I have now recently started to learn more about macroeconometrics (i.e. time series methods) and it kind of appears to me that reverse causality and omitted variable bias kind of get ignored here (I am thinking of 3-4 variable VARs that obvisouly don't capture all the variables that might explain e.g. output or inflation). Is there some mathematical argument that makes these problems disappear magically? Or is macroeconometrics in general less convincing?

I have heard of the credibility revolution in empirical micro and that nothing similar happened in macro, but how much can we trust results from something like an IRF estimation or how convincing are arguments based on IRF results?

PS. I know that in forecasting you can be kind of agnostic to any causality issues, so in this question I am not asking about forecasts. Also I am not putting microeconomics vs macroeconomics in this question. I am only talking about two different econometric approaches.

**Tags:** econometrics, time-series, empirical-evidence, empirical-methods

> **Comments:**
> **username_1**: Most time series analysis depends upon an assumption that statistical properties of tomorrow are like the statistical properties of today, regardless of how parsimonious or how rich your model is in terms of parameters. That doesn't make me a sceptic on econometrics, but instead, it is important to understand it's limitations.  

---

## 1 answers

### Answer by username_2 on 2020-11-04. Score: 3

I do not think that macroeconomic analysis would be 'less trust worthy'. In the same ways as you can solve the issue of omitted variables and reverse causality in microeconomics you can do it macroeconomics as well. Macroeconomics, is not just vector autoregressions (VAR - which by the way corrects for endogeneity). Simultaneous equation techniques, even difference-in-difference (DiD) and regression discontinuity (RD) etc are used even in macro literature even if they are less common (see this recent paper on identification in macroeconomics Nakamura & Steinsson, 2018).

In addition I think you are actually exaggerating quality of quasi-experimental methods such as IV or DiD. For example, this recent paper shows that large portion of IV papers uses weak instruments as the rule of thumb that $F$-stat. should be above 10 is invalid and true threshold is around $F>105$ (see here [Lee McCrary Moreira & Porter 2020](https://arxiv.org/abs/2010.05058)). In addintion DiD are notorious for $p$-hacking and other issues, [Duflo & Mullainathan (2004)](https://academic.oup.com/qje/article/119/1/249/1876068) literally have paper titled: "How much should we trust differences-in-differences estimates?" Spoiler alert their answer is not as much as people usually do.

I think it would be fair to say that a reasonable person could defend saying that typical VAR has probably on average less issues that let's say DiD (but to be clear I am not claiming that as even this lacks nuance as the model selection and specification is always case dependent there is no one model uber ales - even RCTs have a problem when it comes to generalization of results).

When it comes to just empirics in general I could not find any paper that would show that empirical research in macroeconomics is less replicable than in microeconomics. I would also take an issue with claim that macroeconomic models cannot control for all relevant variables. There are macro models (e.g. dynamic stochastic general equilibrium (DSGE)) that include scores of different equations and even more variables. Next, even despite of this when it comes to omitted variable bias you might have a valid point although a lot of attention is being paid to the issue it might sometimes be hard to solve, but with the reverse causality your accusation that macroeconomics does not pay enough attention is unfounded. Most published research in macroeconomics pays very large attention to reverse causality (especially in present day, maybe 50-60 years ago that would be valid point as well). In fact that is why models and estimation techniques like VAR, DSGE, IV, GMM and many other simultaneous equations approaches are so popular and dominate macroeconomics, so this second charge is simply unfounded.

However, when it comes to forecasts I would agree that macroeconomic forecasting is for sure less accurate then microeconomic forecasting (e.g. forecasting aggregate output vs firm forecasting demand for its onw products). However, there the fault would not be in the models themselves necessary but rather with the fact that macroeconomic aggregates contain more noise, are harder to measure, and are reported on lower frequency and of course with less data it is harder to build good forecasting model.

Lastly, it is not really easy to qualify how much you should trust a model in any field. You should approach any model and set of results with healthy doze of skepticism, be they from VAR or RTC or any other approach. Different situations will require different identification strategies. Furthermore, you should not put all faith into single paper but wait for replications/put the paper into larger context of pre-existing literature. I also tried to find some work that would compare replicability of microeconomics vs macroeconomics, but unfortunately there does not seem to be such paper but I would not expect macroeconomics having much worse replicability rates then microeconomics per se.

> **Comments:**
> **username_3**: I love reading your answers and comments because you know more about econometrics than I'll ever know and they're chock full of wisdom.  Then again, that's not a huge compliment because I know very little :) . But why is your focus on reproducibility ? Something can be reproduced and still not be high quality. (i.e., bogus or useless ). Shouldn't the focus, in the end, be forecasting ability ?  
> **username_2**: @markleeds thanks for the compliments, you make me blush :). Now to respond to the comment: when we talk about reproducibility we do not mean just take the same data use the same method and get the same result - of course this can be always done even with p-hacked papers. Reproducibility of a result is often colloquially used even for a situation where someone tries to test the same relationship even with slightly different method the results should be similar, or trying to test the same relationship on different sample and see if results are reasonably robust.  
> **username_2**: @markleeds for example colloquially people often refer to Nuemark and Wascher study that tried to reproduce the minimum wage study of Card and Kruger, using completely different dataset as reproduction study - card and Kruger used self reported data while nuemark and wascher used hard payroll data. If two studies like this agree the result is considered to be reproducible and if not it would not be (at least colloquially in economics - I am not sure if maybe wider statistical literature on uses reproducibility also in more narrow sense.  
> **username_0**: Maybe I did not express this well, I am in no way putting micro over macro theory in this question. Also I know that one can use quasi experimental methods for macroeconomic issues. I am only talking about time series econometrics vs. the quasi experimental stuff. I also know the DiD paper, but I think since 2004 a lot has gotten better (of course nothing is ever good enough though). Lastly, I think the advantage of the quasi exp. methods is that weaknesses can be revealed quite easily, e.g. a good paper will show and discuss how weak/strong an instrument is.  
> **username_2**: @username_0 but good papers will also show how good/bad VAR or error correction model or DSGE is. If you are asking if it is any harder to check whether the models are applied correctly in macro or micro then answer is resounding no. However, I kinda can see why you might have this perception. In order to learn time series analysis you first need to learn cross-sectional analysis so you most likely been exposed to content that covered cross-sectional analysis in great detail but only grazed time series models - which will then be covered in separate classes.  
> **username_3**: @username_2: I got it now. In statistics field, when one refs to reproducibility, they are strictly talking about getting the same EXACT results. Thanks for clarification. Note though that I would argue that your definition of reproducibility is questionable because two papers that reproduce each others results by your definition could still have their own seperate issues. ( questionable assumptions,  whatever ).  
> **username_2**: @markleeds is there in statistics maybe better word for it? I would not mind to also edit that in to make it more clear. And I also agree with you but the idea is that if different approaches with all their flaws/issues manage to get the same/similar results there will likely be some effect there. I mean each result can be sort of think of as an observation like in a meta study. Yes of course there are still issues like publication bias but I would still argue seeing multiple similar results increases the confidence that we are really measuring some true effect  
> **username_3**: @username_2: good point. I basically agree that the more evidence the stronger the evidence. Unfortunately, I don't know what the word is in statistics ? In fact, I'm not sure if there is one word but atleast I understand what your definition of reproducibility is. Unfortunately ( or fortunately ), due to R and github,  reproducibility in statistical academics is getting much hype these days but it has the "duplicate the results" meaning that I mentioned earlier. I'll try to look around when I have some time but there may not be one ? All the best and I'm sorry that I can't help there.  
> **username_4**: @markleeds, what you describe might be the programming rather than statistical understanding of *reproducibility*.  
> **username_3**: Hi Richard: In statistical world, the term means EXACTLY DUPLICATING the statistical results of someone else, given the same data. So, yes, definitely programming ( R is kind of standard in stat world )  is the key to it if that's what you meant.  
> **username_2**: @RichardHardy I am tagging you since mark forgot to. If it would not be bothersome could you respond? I would be interested in getting to the bottom of this semantic dispute  
> **username_2**: @markleeds I found this answer on cross-validated it says reproducibility can have both meanings - one is considered to be narrow meaning and one is considered to be broad meaning https://stats.stackexchange.com/a/15006/89502 is it possible that you and Richard just focus on one of these different views  
> **username_3**: @username_2: I didn't know that there were two views ( on the meaning in statistics anyway. I can see how there would be two views in general ) so my apologies. Maybe your view was in statistics in previous years and then, say in the last decade or so, the one I was talking about came around, so they're both around.   I'll check out the link. My mistake and sorry to lead you astray if I did.  
> **username_3**: @username_2: I read the link and using reproducible "analysis" or "data analysis" etc does make sense. It would make the meaning clearer for sure.  

---

