<p>The answer to your question:</p>
<blockquote>
<p>I wanted to know if it is possible to have an exhaustive set of things I need to test to be reasonably confident about the p-values I get, both overall and for each specific regressor's coefficient</p>
</blockquote>
<p>is &quot;no&quot;, but for a different reason than is stated in the comments.</p>
<p>To start this, let me ground you in what a p-value is and what it is not.  First, it is <em>not</em> a probability statement.  It is a conditional frequency statement.  It is conditional on a utility function and a model of the world.  If you have a bad model of the world, then your p-values are suspect.</p>
<p>Frequentist tests, as opposed to Bayesian tests, attempt to show that some null hypothesis is false.  It does so by assuming that the null is true.  That is not a trivial statement for two reasons.  First, the null hypothesis acts as information.  You are conditioning all the observations on that assumptions, as if true.  If you would change your null, you would change the inference on your assumptions.  Second, there is no way to distinguish a chance effect from a false null.</p>
<p>A p-value is a statement that if you would repeat your experiment, possibly an infinite number of times, then the frequency of seeing the result you saw or a result that is more extreme is no more common than the stated p-value <em>if the null model is the true model.</em></p>
<p>So, let us look at the things you need to get right for a p-value to imply its abstract meaning.</p>
<p>First, your utility function needs to be correct.  Least squares models assume that losses are quadratic, that is to say <span class="math-container">$$U(\hat{\theta},\theta)=-c(\hat{\theta}-\theta)^2.$$</span>  If you change your model, then you change your estimator, which implies that your choice of p-values changes.</p>
<p>For example, if you would assume that <span class="math-container">$U(\hat{\theta},\theta)=-c|\hat{\theta}-\theta|$</span>, then you would end up with median estimators and would use things such as Theil's or Quantile regression.  The results are more robust but less powerful.</p>
<p>Second, you need to get your specification correct.  P-values assume the null model is true, if you have a misspecified model, then your p-values will be suspect, although not automatically so far from reality as to discard them.</p>
<p>Third, you need to match the correct assumptions to the real null model.  This is the danger in testing for heteroskedasticity.  If you have heteroskedasticity, but it fails to reject the null, then you have a bad set of assumptions.  If you do not have heteroskedasticity, but it rejects the null, then you have a bad set of assumptions.</p>
<p>The actual fix for problems like that is to approach the assumptions logically.  That is why there is no single book or reference set for this problem.</p>
<p>Diamond prices are heteroskedastic in size but in a weird way.  Simple logic would tell you that they are probably heteroskedastic with respect to size.</p>
<p>Very small diamonds have only two purposes, industrial uses such as drilling, and as additions to jewelry.  As jewelry, their price is very high.  As an industrial material, the price is very low.</p>
<p>Most medium-sized diamonds are either close to their desired size for jewelry or need cut into very small pieces for industrial use.  However, the bulk of their use will be to make medium-sized diamonds cut to specification.  It will be easier to estimate their value as is.</p>
<p>Very large diamonds might remain highly valued, very large diamonds.  Think of the Hope Diamond as an example.  Others may make hundreds of small diamonds for industrial use.</p>
<p>The result is that you get a bowtie version of heteroskedasticity.  Small diamonds and very large diamonds have enormous price variability.  Medium diamonds are quite predictable as to the final cut and value and so are priced in a much narrower range.</p>
<p>You should never test for heteroskedasticity with diamonds because there is a way to think this through.  Testing should be reserved for things that you have no subject matter knowledge regarding.</p>
<p>Most p-values are robust because most Frequentist estimators are minimax estimators, that is to say, the estimator minimizes the maximum amount of risk that you must take under a specified utility function.</p>
<p>As a result, p-values tend to be conservative in their use of information and any resulting inference.</p>
<p>The reason that there is no one list is that if you can assert something about the population as true, such as diamond prices being logically heteroskedastic in size, then you can condition your model with that knowledge.</p>
<p>As a result, if you were to test diamond prices and they tested as homoskedastic, then you could ignore the test result as irrelevant.</p>
<p>There isn't a list.  Instead, you need to determine what researchers in a narrow domain do.</p>
<p>Let me give you one more example to see why that matters.</p>
<p>If a brick-and-mortar retailer needs to alter its revenue, it can issue coupons. It has the ability to control its revenue and its sales volume.</p>
<p>Electrical utilities, generally, cannot control their volume or price.  Even if they could mail coupons to use additional electricity, how would anyone monitor this sale?  Electric utilities control their costs, not their revenue.</p>
<p>That discussion alone tells you that the accounting information that management sees as important for the marginal decision is different.  If you give them a common specification, then your model will be misspecified.  Your p-values are off.  You omitted a variable, industry type.</p>
<p>Finding correct p-values are about more than overcoming technical obstacles, such as multicollinearity and heteroskedasticity, it is about understanding your domain of research.</p>
<p>Fortunately, p-values are usually robust.  If you set <span class="math-container">$\alpha=.05$</span> and <span class="math-container">$p&lt;.01$</span> then you are probably safe.  On the other hand, if <span class="math-container">$p&lt;.04999$</span> and your work is a bit shaky, then do not feel confident even if you followed the technical rules.</p>
<p>There is no one book, no simple list.  There are just domains where researchers discover unexpected things and correct for them.</p>

CC BY-SA 4.0
2021-02-02T18:27:53.333Z
2021-02-02T18:27:53.333Z
Dave Harris
<p>I have learned the basics of the Classical Linear Regression Model and also various diagnostic tests to check if the assumptions of the CLRM are met, such as homoskedadticity, absence of near perfect multicollinearity, normality of residuals etc.</p>
<p>I wanted to know if it is possible to have an exhaustive set of things I need to test to be reasonably confident about the p-values I get, both overall and for each specific regressor's coefficient (in a multiple regression with cross sectional data).</p>
<p>For example, what are the things that a researcher would test to make sure that the p-values are technically correct (or at least acceptable) and publishable in an academic journal (not talking about the usefulness of the topic or whether there is causality etc, just the statistical inference, if that is acceptable or not)? Different textbooks/ lectures I have seen have put differing amounts of emphasis on different things. I wanted to know what would a professional researcher do?</p>

CC BY-SA 4.0
2021-02-02T11:06:40.930Z
I think this is beyond scope of single se answer. OLS can be applied for either cross-section, time series and panel data all which will have their unique challenges. It would require several pages to provide brief exhaustive overview of that. In addition, even though you write OLS, reading the question I have feeling that by OLS you mean linear regression models - since OLS (narrowly defined) is relatively less used in research nowadays outside randomized control trials - which would make it unreasonably broad.
1muflon1
CC BY-SA 4.0
2021-02-02T11:30:00.893Z
Thank you for responding! Can you please suggest some book that exhaustively covers linear regression models for cross-sectional data, especially how it is actually applied in empirical research? Reading from introductory textbooks, I fear if miss out some important condition to check that makes my conclusions invalid
Ishan Kashyap Hazarika
CC BY-SA 4.0
2021-02-02T11:35:47.437Z
Nowadays this is so broad that often detailed books dont cover everything but only portion of it. For example, Stachurski A Primer in Econometric Theory, covers some basic cross-sectional and time series models in excruciating detail - literary deriving them from first principles, so every single detail that could affect estimation is mentioned, but at a cost of focusing only at few models. Hamilton Time series analysis is also deep but has narrow focus on time series models. Pesaran Time Series and Panel Data Econometrics is more broader but also slightly less deep.
1muflon1
CC BY-SA 4.0
2021-02-02T11:44:59.177Z
My understanding is that, in practice, hardly anyone conducts these tests; and nobody reports the results of such tests. e.g. people never check for heteroskedasticity; they just report heteroskedasticity robust standard errors
afreelunch
CC BY-SA 4.0
2021-02-02T13:57:50.657Z
@afreelunch that is nonsense - if you got that understanding then you must have experience with extremely unscrupulous scholars and people who do low quality research. I worked for various central banks in the EU in the past as research assistants and we always did these checks. Also at our university department where I now do PhD I don’t know anyone who would skip these. When working for professors they always require their assistants to do them and it is not uncommon to get questions about whether people making auxiliary tests for this or that issue at conferences and discussing that in Q&A
1muflon1
CC BY-SA 4.0
2021-02-02T14:04:05.880Z
they wont be reported in papers due to page limits. Even despite of this you can actually find them in supplied code and even if they are not there its usually not because they were not done but because people delete them out of spite to provide as little code for competition as possible while fulfilling only minimum journal requirement on code disclosure...
1muflon1
CC BY-SA 4.0
2021-02-02T15:59:18.147Z
@1muflon1 The reason why these tests are not conducted is that they are unnecessary. As I have already noted, it is rare to check for heteroskedasticity every time you run a regression since the (current) practice is to use robust standard errors. Similarly, it is rare to justify regressions by appealing to normally distributed errors since one can appeal to CLT anyway
afreelunch
CC BY-SA 4.0
2021-02-02T16:01:01.643Z
@1muflon1 Having said all of this, your extremely confident tone suggests that maybe you know that something that I don't! So perhaps you can link some of these papers which check for heteroskedasticity and normality every time they run a regression? (Well published papers please, not whatever you wrote for a central bank!)
afreelunch
CC BY-SA 4.0
2021-02-02T16:15:51.147Z
@afreelunch you don’t do that every single time you run regression, as that would be unnecessary but you also do not apply robust errors haphazardly. It is common to first check whether either only heteroskedasticity or autocorrelation or both are problem in most general model apply correct errors (either white or HAC) and then you of course won’t repeat the testing for every single robustness check where you just drop some control variables... in addition when dealing with panel data further testing for cross-sectional dependence is commonplace. Non-linearity tests and multicollinearity tests
1muflon1
CC BY-SA 4.0
2021-02-02T16:18:45.543Z
Are also common but of course only if plot of residuals reveals some strange patterns, or if you see your model having high F-stat and low t statistics. Unit root testing is quite commonplace for variables where presence of unit roots is ambiguous and not yet well determined in the literature. I could go on and on. Of course people don’t waste time applying these tests repeatedly to every robustness check as that is pointless and in addition of course if there is no evidence for some problems such as non-linearity or multicollinearity or similar issues tests are skipped but that does not mean
1muflon1
CC BY-SA 4.0
2021-02-02T16:21:06.003Z
testing is not done. If you want to see an example of published paper with some vestigial error diagnostics left in the code supplied to the journal you can have look at Bringing education to Afghan girls by Burda and Linden
1muflon1
CC BY-SA 4.0
2021-02-02T16:26:03.760Z
individuals "do not apply robust errors haphazardly." This is wrong: it is standard to *always* use robust standard errors (which remain valid even in the presence of heteroskedasticity)
afreelunch
CC BY-SA 4.0
2021-02-02T16:27:08.687Z
Regarding tests for normality, I suggest you Google 'central limit theorem'
afreelunch
CC BY-SA 4.0
2021-02-02T16:31:41.420Z
@afreelunch no 1. First of all there are multiple robust errors. 2. If you actually read literature on robust errors you will see that for example if autocorrelation is not present but you use Newey-West (HAC) errors variances will be unnecessarily more inflated than when you just use white errors that might occasionally lead to wrong inference in marginal cases. In addition, I have good understanding of central limit theorem one of my Bc was in pure econometrics. I would rather advise you to read some econometrics textbook or use google scholar to read literature on robust errors
1muflon1
CC BY-SA 4.0
2021-02-02T16:41:43.283Z
@afreelunch Also on the note of normality if you would actually checked the econometric literature you would see it is not that simple. There are multiple issues that will lead to violation of normal distribution of residuals even in the presence of central limit theorem - also in any econometrics graduate textbook you could find that central limit theorem does not say anything about the distribution of residuals in a statistical model - it talks about their **asymptotic distribution**. For example, ADF test that is based on OLS at its heart will generally not have normally distributed errors
1muflon1
CC BY-SA 4.0
2021-02-02T17:04:47.730Z
Also further on the whole note of regression diagnostic - it is simply not correct to state that it’s not done - especially when talking about state of the art research and good uni departments and policy institutes. However, of course it is not done in a way that a researcher just applies every single test possible to the model. In the same way as medical doctor will first look at a patient and then recommends tests based on medical literature and not all possible medical test a good econometrician will start with some preliminary model and uses knowledge of lit. to decide what needs testing
1muflon1
<econometrics><regression><applied-econometrics><research>
Making sure the p-values of my OLS estimates are correct
Ishan Kashyap Hazarika
# Making sure the p-values of my OLS estimates are correct

(Asked by: username_0 on 2021-02-02)

I have learned the basics of the Classical Linear Regression Model and also various diagnostic tests to check if the assumptions of the CLRM are met, such as homoskedadticity, absence of near perfect multicollinearity, normality of residuals etc.

I wanted to know if it is possible to have an exhaustive set of things I need to test to be reasonably confident about the p-values I get, both overall and for each specific regressor's coefficient (in a multiple regression with cross sectional data).

For example, what are the things that a researcher would test to make sure that the p-values are technically correct (or at least acceptable) and publishable in an academic journal (not talking about the usefulness of the topic or whether there is causality etc, just the statistical inference, if that is acceptable or not)? Different textbooks/ lectures I have seen have put differing amounts of emphasis on different things. I wanted to know what would a professional researcher do?

**Tags:** econometrics, regression, applied-econometrics, research

> **Comments:**
> **username_1**: I think this is beyond scope of single se answer. OLS can be applied for either cross-section, time series and panel data all which will have their unique challenges. It would require several pages to provide brief exhaustive overview of that. In addition, even though you write OLS, reading the question I have feeling that by OLS you mean linear regression models - since OLS (narrowly defined) is relatively less used in research nowadays outside randomized control trials - which would make it unreasonably broad.  
> **username_0**: Thank you for responding! Can you please suggest some book that exhaustively covers linear regression models for cross-sectional data, especially how it is actually applied in empirical research? Reading from introductory textbooks, I fear if miss out some important condition to check that makes my conclusions invalid  
> **username_1**: Nowadays this is so broad that often detailed books dont cover everything but only portion of it. For example, Stachurski A Primer in Econometric Theory, covers some basic cross-sectional and time series models in excruciating detail - literary deriving them from first principles, so every single detail that could affect estimation is mentioned, but at a cost of focusing only at few models. Hamilton Time series analysis is also deep but has narrow focus on time series models. Pesaran Time Series and Panel Data Econometrics is more broader but also slightly less deep.  
> **username_2**: My understanding is that, in practice, hardly anyone conducts these tests; and nobody reports the results of such tests. e.g. people never check for heteroskedasticity; they just report heteroskedasticity robust standard errors  
> **username_1**: @username_2 that is nonsense - if you got that understanding then you must have experience with extremely unscrupulous scholars and people who do low quality research. I worked for various central banks in the EU in the past as research assistants and we always did these checks. Also at our university department where I now do PhD I don’t know anyone who would skip these. When working for professors they always require their assistants to do them and it is not uncommon to get questions about whether people making auxiliary tests for this or that issue at conferences and discussing that in Q&A  
> **username_1**: they wont be reported in papers due to page limits. Even despite of this you can actually find them in supplied code and even if they are not there its usually not because they were not done but because people delete them out of spite to provide as little code for competition as possible while fulfilling only minimum journal requirement on code disclosure...  
> **username_2**: @username_1 The reason why these tests are not conducted is that they are unnecessary. As I have already noted, it is rare to check for heteroskedasticity every time you run a regression since the (current) practice is to use robust standard errors. Similarly, it is rare to justify regressions by appealing to normally distributed errors since one can appeal to CLT anyway  
> **username_2**: @username_1 Having said all of this, your extremely confident tone suggests that maybe you know that something that I don't! So perhaps you can link some of these papers which check for heteroskedasticity and normality every time they run a regression? (Well published papers please, not whatever you wrote for a central bank!)  
> **username_1**: @username_2 you don’t do that every single time you run regression, as that would be unnecessary but you also do not apply robust errors haphazardly. It is common to first check whether either only heteroskedasticity or autocorrelation or both are problem in most general model apply correct errors (either white or HAC) and then you of course won’t repeat the testing for every single robustness check where you just drop some control variables... in addition when dealing with panel data further testing for cross-sectional dependence is commonplace. Non-linearity tests and multicollinearity tests  
> **username_1**: Are also common but of course only if plot of residuals reveals some strange patterns, or if you see your model having high F-stat and low t statistics. Unit root testing is quite commonplace for variables where presence of unit roots is ambiguous and not yet well determined in the literature. I could go on and on. Of course people don’t waste time applying these tests repeatedly to every robustness check as that is pointless and in addition of course if there is no evidence for some problems such as non-linearity or multicollinearity or similar issues tests are skipped but that does not mean  
> **username_1**: testing is not done. If you want to see an example of published paper with some vestigial error diagnostics left in the code supplied to the journal you can have look at Bringing education to Afghan girls by Burda and Linden  
> **username_2**: individuals "do not apply robust errors haphazardly." This is wrong: it is standard to *always* use robust standard errors (which remain valid even in the presence of heteroskedasticity)  
> **username_2**: Regarding tests for normality, I suggest you Google 'central limit theorem'  
> **username_1**: @username_2 no 1. First of all there are multiple robust errors. 2. If you actually read literature on robust errors you will see that for example if autocorrelation is not present but you use Newey-West (HAC) errors variances will be unnecessarily more inflated than when you just use white errors that might occasionally lead to wrong inference in marginal cases. In addition, I have good understanding of central limit theorem one of my Bc was in pure econometrics. I would rather advise you to read some econometrics textbook or use google scholar to read literature on robust errors  
> **username_1**: @username_2 Also on the note of normality if you would actually checked the econometric literature you would see it is not that simple. There are multiple issues that will lead to violation of normal distribution of residuals even in the presence of central limit theorem - also in any econometrics graduate textbook you could find that central limit theorem does not say anything about the distribution of residuals in a statistical model - it talks about their **asymptotic distribution**. For example, ADF test that is based on OLS at its heart will generally not have normally distributed errors  
> **username_1**: Also further on the whole note of regression diagnostic - it is simply not correct to state that it’s not done - especially when talking about state of the art research and good uni departments and policy institutes. However, of course it is not done in a way that a researcher just applies every single test possible to the model. In the same way as medical doctor will first look at a patient and then recommends tests based on medical literature and not all possible medical test a good econometrician will start with some preliminary model and uses knowledge of lit. to decide what needs testing  

---

## 1 answers

### Answer by username_3 on 2021-02-02. Score: 2 | Accepted answer

The answer to your question:

> I wanted to know if it is possible to have an exhaustive set of things I need to test to be reasonably confident about the p-values I get, both overall and for each specific regressor's coefficient

is "no", but for a different reason than is stated in the comments.

To start this, let me ground you in what a p-value is and what it is not. First, it is *not* a probability statement. It is a conditional frequency statement. It is conditional on a utility function and a model of the world. If you have a bad model of the world, then your p-values are suspect.

Frequentist tests, as opposed to Bayesian tests, attempt to show that some null hypothesis is false. It does so by assuming that the null is true. That is not a trivial statement for two reasons. First, the null hypothesis acts as information. You are conditioning all the observations on that assumptions, as if true. If you would change your null, you would change the inference on your assumptions. Second, there is no way to distinguish a chance effect from a false null.

A p-value is a statement that if you would repeat your experiment, possibly an infinite number of times, then the frequency of seeing the result you saw or a result that is more extreme is no more common than the stated p-value *if the null model is the true model.*

So, let us look at the things you need to get right for a p-value to imply its abstract meaning.

First, your utility function needs to be correct. Least squares models assume that losses are quadratic, that is to say $$U(\hat{\theta},\theta)=-c(\hat{\theta}-\theta)^2.$$ If you change your model, then you change your estimator, which implies that your choice of p-values changes.

For example, if you would assume that $U(\hat{\theta},\theta)=-c|\hat{\theta}-\theta|$, then you would end up with median estimators and would use things such as Theil's or Quantile regression. The results are more robust but less powerful.

Second, you need to get your specification correct. P-values assume the null model is true, if you have a misspecified model, then your p-values will be suspect, although not automatically so far from reality as to discard them.

Third, you need to match the correct assumptions to the real null model. This is the danger in testing for heteroskedasticity. If you have heteroskedasticity, but it fails to reject the null, then you have a bad set of assumptions. If you do not have heteroskedasticity, but it rejects the null, then you have a bad set of assumptions.

The actual fix for problems like that is to approach the assumptions logically. That is why there is no single book or reference set for this problem.

Diamond prices are heteroskedastic in size but in a weird way. Simple logic would tell you that they are probably heteroskedastic with respect to size.

Very small diamonds have only two purposes, industrial uses such as drilling, and as additions to jewelry. As jewelry, their price is very high. As an industrial material, the price is very low.

Most medium-sized diamonds are either close to their desired size for jewelry or need cut into very small pieces for industrial use. However, the bulk of their use will be to make medium-sized diamonds cut to specification. It will be easier to estimate their value as is.

Very large diamonds might remain highly valued, very large diamonds. Think of the Hope Diamond as an example. Others may make hundreds of small diamonds for industrial use.

The result is that you get a bowtie version of heteroskedasticity. Small diamonds and very large diamonds have enormous price variability. Medium diamonds are quite predictable as to the final cut and value and so are priced in a much narrower range.

You should never test for heteroskedasticity with diamonds because there is a way to think this through. Testing should be reserved for things that you have no subject matter knowledge regarding.

Most p-values are robust because most Frequentist estimators are minimax estimators, that is to say, the estimator minimizes the maximum amount of risk that you must take under a specified utility function.

As a result, p-values tend to be conservative in their use of information and any resulting inference.

The reason that there is no one list is that if you can assert something about the population as true, such as diamond prices being logically heteroskedastic in size, then you can condition your model with that knowledge.

As a result, if you were to test diamond prices and they tested as homoskedastic, then you could ignore the test result as irrelevant.

There isn't a list. Instead, you need to determine what researchers in a narrow domain do.

Let me give you one more example to see why that matters.

If a brick-and-mortar retailer needs to alter its revenue, it can issue coupons. It has the ability to control its revenue and its sales volume.

Electrical utilities, generally, cannot control their volume or price. Even if they could mail coupons to use additional electricity, how would anyone monitor this sale? Electric utilities control their costs, not their revenue.

That discussion alone tells you that the accounting information that management sees as important for the marginal decision is different. If you give them a common specification, then your model will be misspecified. Your p-values are off. You omitted a variable, industry type.

Finding correct p-values are about more than overcoming technical obstacles, such as multicollinearity and heteroskedasticity, it is about understanding your domain of research.

Fortunately, p-values are usually robust. If you set $\alpha=.05$ and $p<.01$ then you are probably safe. On the other hand, if $p<.04999$ and your work is a bit shaky, then do not feel confident even if you followed the technical rules.

There is no one book, no simple list. There are just domains where researchers discover unexpected things and correct for them.


---

