<p>There are numerous directions to go which start moving you beyond ordinary least squares (OLS), linear regression. The universe of statistical methods is large! </p>

<p>Two books that I particularly enjoyed are <em>Econometrics</em> by Hayashi and <em>Elements of Statistical Learning</em> by Hastie et. al. Looking back at your question, these books may be too advanced. But maybe not. An easier version of the latter is <em>An Introduction to Statistical Learning</em> (and may be interesting from the perspective of a broader exposure to data science than just econometrics).</p>

<ul>
<li>Hayashi's <em>Econometrics</em> introduces a variety of methods through the lens of GMM and with an eye towards time-series econometrics.</li>
<li><em>Elements of Statistical Learning</em> is a modern classic of the statistics, machine learning literature. It's great for opening your eyes to methods outside of traditional econometrics.</li>
</ul>

<h3>Some examples beyond ordinary least squares...</h3>

<ul>
<li><p><a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" rel="nofollow noreferrer">Maximum likelihood estimation (MLE)</a></p>

<p>You must know this if you continue doing statistics. It's a ubiquitous workhorse.</p>

<p>If you can specify the <a href="https://en.wikipedia.org/wiki/Likelihood_function" rel="nofollow noreferrer">likelihood function</a> then parameters of the likelihood function can be estimated by maximizing the likelihood function. In certain special cases, (eg. linear regression with conditionally normal error terms) the OLS estimator is the MLE estimator. You've undoubtedly encountered MLE estimation before if you've estimated a <a href="https://en.wikipedia.org/wiki/Logistic_regression" rel="nofollow noreferrer">logit model</a>. MLE is all over physics, engineering, and the sciences.</p>

<p>There are issues with applying MLE to economic models though. Often we know that an overall economic model is false. A model will imply certain facts that are entirely fallacious. Forcing the model to match the data in a maximum likelihood sense may not induce a useful choice of parameters. GMM can be used as an alternative approach to selectively test certain predictions of a model.</p></li>
<li><p><a href="https://en.wikipedia.org/wiki/Generalized_method_of_moments" rel="nofollow noreferrer">GMM</a> is another broad method for estimating parameters based upon moment conditions that in expectation should be zero. Hayashi's book <em>Econometrics</em> develops ordinary least squares regression, instrumental variables, maximum likelihood, and other methods as special cases of GMM with different moment conditions. OLS can be thought of as GMM using the orthogonality condition of the regressors and the error terms. MLE can be derived as GMM on the <a href="https://en.wikipedia.org/wiki/Score_(statistics)" rel="nofollow noreferrer">score</a>.</p>

<p><a href="https://johnhcochrane.blogspot.com/2013/10/hansens-nobel.html" rel="nofollow noreferrer">A John Cochrane ode to GMM is here.</a></p></li>
<li><p><a href="https://en.wikipedia.org/wiki/Matching_(statistics)" rel="nofollow noreferrer">Matching methods</a> for estimating causal effects are common in certain areas of economics.</p>

<p>The idea is to match a treated entity with an untreated entity based upon observable characteristics. A widely used technique for example is <a href="https://en.wikipedia.org/wiki/Propensity_score_matching" rel="nofollow noreferrer">propensity score matching</a></p></li>
<li><p>There are all kinds of variations on classic linear methods:</p>

<p>The idea here is to start with ordinary least squares but then to bias coefficient estimates towards zero to reduce overfitting and improve out of sample prediction.</p>

<ul>
<li><a href="https://en.wikipedia.org/wiki/Tikhonov_regularization" rel="nofollow noreferrer">Ridge regression</a></li>
<li><a href="https://en.wikipedia.org/wiki/Lasso_(statistics)" rel="nofollow noreferrer">Lasso</a></li>
</ul></li>
</ul>

<h3>References</h3>

<p><a href="https://rads.stackoverflow.com/amzn/click/0691010188" rel="nofollow noreferrer">Fumio, Hayashi, 2000, <em>Econometrics</em></a></p>

<p><a href="https://rads.stackoverflow.com/amzn/click/B00475AS2E" rel="nofollow noreferrer">Hastie, Trevor, Robert Tibshirani, Jerome Friedman, 2009, <em>Elements of Statistical Learning</em></a></p>

<p><a href="https://rads.stackoverflow.com/amzn/click/1461471370" rel="nofollow noreferrer">James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2017, <em>An Introduction to Statistical Learning</em></a></p>

CC BY-SA 3.0
2017-12-03T11:03:31.023Z
Thanks! I read Introduction to statistical learning and mainly motivation for this question came from disappointment of not seeing any regularization methods used in our economics and econometrics classes...thanks :)
econ
CC BY-SA 3.0
2017-12-04T18:00:20.623Z
@econ You're going to see more machine learning techniques used in economics, especially for problems or subproblems where the interest is in a forecast $\hat{y}$ and you don't care as much why or how you got that forecast. Be aware though that many machine learning techniques can be problematic for $\hat{\beta}$ problems where you're trying to estimate a specific parameter (eg. what's the causal effect of $x$). Machine learning techniques etc... may also provoke hostile questions from many economists who don't know the techniques and/or view them with suspicion.
Matthew Gunn
CC BY-SA 3.0
2017-11-28T22:47:27.577Z
2017-11-30T16:32:40.490Z
2017-11-30T16:32:40.490Z
Matthew Gunn
<p>Linear regression, despite its simplicity, it is actually a very powerful tool. That's why it's everywhere in econometrics, to give you an example you're maybe familiar with, consider an <a href="https://en.wikipedia.org/wiki/Autoregressive_model" rel="nofollow noreferrer">auto-regressive</a> model, turns out you can write the future state of a variable that follows this model as a linear combination of previous states</p>

<p>$$
X_t = C + \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \eta_t \tag{1}
$$</p>

<p>so, if you know the weights $\{\phi_k \}_{k=1}^p$ you can predict what's going to be the future values of $X$. The interesting part is that these numbers are obtained through linear regression: just call $y = X_t$, $x_1 = X_{t-1}, x_2 = X_{t-2},\cdots$ and realize Eq. (1) can be written as </p>

<p>$$
y = C + \phi_1 x_1  + \phi_2 x_2 + \cdots + \phi_p x_p + \epsilon \tag{2}
$$</p>

<p>In this sense linear regression can be used for forecasting. But there are other tools, I will link you to <a href="https://stats.stackexchange.com/questions/10162/how-to-apply-neural-network-to-time-series-forecasting">this</a> other thread where you can get an idea on how <em>neuronal networks</em> can be used for this task: forecasting time series. But forecasting can also be done with a plethora of methods: <a href="http://www.svms.org/finance/" rel="nofollow noreferrer">support vector machines</a> are popular choices.</p>

CC BY-SA 3.0
2017-11-25T23:07:17.723Z
2017-11-25T23:07:17.723Z
caverac
<p>You can always think of a linear regression model as a first-order <a href="https://en.wikipedia.org/wiki/Taylor_series#Approximation_error_and_convergence" rel="nofollow noreferrer">Taylor approximation</a> of some nonlinear regression model. So linearity itself is not a particularly serious issue. </p>

<p>That being said, there are a couple of common nonlinear regression models that are accessible at the undergraduate level, namely, models with binary dependent variables: <a href="https://liberalarts.utexas.edu/prc/_files/cs/Fall2013_Moore_Logistic_Probit_Regression.pdf" rel="nofollow noreferrer">logit and probit</a>.</p>

CC BY-SA 3.0
2017-11-27T00:19:29.880Z
2017-11-27T00:19:29.880Z
Herr K.
<p>Regression is the mapping of <em>any</em> function of any dimension onto a result.  There exists an infinite number of functions.  Furthermore, there is a wider range of <em>linear</em> regression tools than just least squares style solutions.  My guess is that you have yet to even come close to covering the linear tools if you are in your third year.</p>

<p>For more linear tools, look at quantile regression and Theil's regression.  Both are very robust.  Quantile, ordinary least squares and Theil's method of regression are usable for a polynomial of any degree.  If you are studying least squares style methods such as OLS, GLS or FGLS then you are also studying non-linear methods at the same time.  All linear tools can be readily adapted to non-linear problems.  The part no one has told you is that you are studying non-linear regression, but using linear models to study the properties because they are easier.</p>

<p>The linkage is similar to the linkage between Algebra 2 in high school and calculus 1.  Grounding in the former is necessary for the latter.</p>

<p>Instead of worrying about "non-linear" methods, I would recommend taking two different routes.</p>

<p>The first is non-parametric and distribution-free methods.  The second is Bayesian methods.  Your instructor will hate me forever for this recommendation.</p>

<p>Distribution-free methods are the easiest to understand.  They are robust under any distributional assumption, which results in a material loss of power.  They always work, but they are weak solutions because you know so little about how the world works.  Theil's regression is such an example.</p>

<p>Non-parametric methods are a little harder to understand.  They do not depend upon a parameter to perform inference.  So, for example, when you have performed a t-test, you have assumed that a mean exists and that it is meaningful.  It is not always true that a mean exists in a distribution and it is not always true that it is a meaningful measurement when it does exist.  Non-parametric methods allow you to perform tests on data without reference to a parameter.  As with distribution-free methods, they are weaker than the equivalent parametric test.  They always work, but they are more likely to not detect an effect that is really there.</p>

<p>Finally, after you have looked at distribution-free and non-parametric methods, then you should look at Bayesian methods.  Bayesian methods are older than Frequentist methods, but allow you to solve problems that have no Frequentist solution.  On the surface, they can look just like the problems you are solving now, but below the surface, they open up entire world's of prediction and modeling not available with Frequentist methods.</p>

<p>Bayesian methods reverse the direction of uncertainty.  With a null hypothesis method, you assert that the null is true and use the data to falsify it.  Essentially you are performing a test as $\Pr(x|\theta)$, that is to say, what is the probability of seeing this data if the null is actually true.  The use of Bayesian methods reverses the question.  The Bayesian asks $\Pr(\theta|x)$?  The Bayesian method asks, "what is the probability the hypothesis is true, given the data that was actually seen?"</p>

<p>The Frequentist works in the "sample space," which is the set of all possible outcomes of a random event.  The Bayesian works in the "parameter space" which is the set of all possible explanations.</p>

<p>A good post showing the differences that you can readily see is on the difference between a Frequentist confidence interval and a Bayesian credible interval.  It is at <a href="https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval">https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval</a></p>

<p>William Bolstad writes a good introductory book on Bayesian methods if you have had calculus through integration.  You cannot do Bayesian methods without knowing integration.</p>

<p>There is a giant world out there.  Go explore.</p>

CC BY-SA 3.0
2017-11-26T15:50:43.523Z
2017-11-26T18:33:40.270Z
2017-11-26T18:33:40.270Z
Dave Harris
<p>I'm third year economics student and all econometrics we had so far and basically all empirical studies in economic subjects we had so far are linear regression. Is there any alternative, can anyone suggest any reading material or direction in which I can explore? </p>

CC BY-SA 3.0
2017-11-25T11:24:22.020Z
It strongly depends on what you want to do: forecasting? classification?
caverac
CC BY-SA 3.0
2017-11-25T13:37:51.457Z
Can you give me example for both?
econ
CC BY-SA 3.0
2017-11-25T17:03:07.963Z
By linear regression, do you mean linear in the parameters or linear functional form?  There is an explanation of the difference at http://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-difference-between-linear-and-nonlinear-equations-in-regression-analysis
Adam Bailey
CC BY-SA 3.0
2017-11-25T20:35:58.157Z
Linear functional form.
econ
<regression>
Alternative to linear regression
econ
# Alternative to linear regression

(Asked by: username_0 on 2017-11-25)

I'm third year username_0omics student and all username_0ometrics we had so far and basically all empirical studies in username_0omic subjects we had so far are linear regression. Is there any alternative, can anyone suggest any reading material or direction in which I can explore?

**Tags:** regression

> **Comments:**
> **username_1**: It strongly depends on what you want to do: forecasting? classification?  
> **username_0**: Can you give me example for both?  
> **username_2**: By linear regression, do you mean linear in the parameters or linear functional form?  There is an explanation of the difference at http://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-difference-between-linear-and-nonlinear-equations-in-regression-analysis  
> **username_0**: Linear functional form.  

---

## 4 answers

### Answer by username_5 on 2017-11-26. Score: 3 | Accepted answer

Regression is the mapping of *any* function of any dimension onto a result. There exists an infinite number of functions. Furthermore, there is a wider range of *linear* regression tools than just least squares style solutions. My guess is that you have yet to even come close to covering the linear tools if you are in your third year.

For more linear tools, look at quantile regression and Theil's regression. Both are very robust. Quantile, ordinary least squares and Theil's method of regression are usable for a polynomial of any degree. If you are studying least squares style methods such as OLS, GLS or FGLS then you are also studying non-linear methods at the same time. All linear tools can be readily adapted to non-linear problems. The part no one has told you is that you are studying non-linear regression, but using linear models to study the properties because they are easier.

The linkage is similar to the linkage between Algebra 2 in high school and calculus 1. Grounding in the former is necessary for the latter.

Instead of worrying about "non-linear" methods, I would recommend taking two different routes.

The first is non-parametric and distribution-free methods. The susername_0d is Bayesian methods. Your instructor will hate me forever for this recommendation.

Distribution-free methods are the easiest to understand. They are robust under any distributional assumption, which results in a material loss of power. They always work, but they are weak solutions because you know so little about how the world works. Theil's regression is such an example.

Non-parametric methods are a little harder to understand. They do not depend upon a parameter to perform inference. So, for example, when you have performed a t-test, you have assumed that a mean exists and that it is meaningful. It is not always true that a mean exists in a distribution and it is not always true that it is a meaningful measurement when it does exist. Non-parametric methods allow you to perform tests on data without reference to a parameter. As with distribution-free methods, they are weaker than the equivalent parametric test. They always work, but they are more likely to not detect an effect that is really there.

Finally, after you have looked at distribution-free and non-parametric methods, then you should look at Bayesian methods. Bayesian methods are older than Frequentist methods, but allow you to solve problems that have no Frequentist solution. On the surface, they can look just like the problems you are solving now, but below the surface, they open up entire world's of prediction and modeling not available with Frequentist methods.

Bayesian methods reverse the direction of uncertainty. With a null hypothesis method, you assert that the null is true and use the data to falsify it. Essentially you are performing a test as $\Pr(x|\theta)$, that is to say, what is the probability of seeing this data if the null is actually true. The use of Bayesian methods reverses the question. The Bayesian asks $\Pr(\theta|x)$? The Bayesian method asks, "what is the probability the hypothesis is true, given the data that was actually seen?"

The Frequentist works in the "sample space," which is the set of all possible outcomes of a random event. The Bayesian works in the "parameter space" which is the set of all possible explanations.

A good post showing the differences that you can readily see is on the difference between a Frequentist confidence interval and a Bayesian credible interval. It is at <https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval>

William Bolstad writes a good introductory book on Bayesian methods if you have had calculus through integration. You cannot do Bayesian methods without knowing integration.

There is a giant world out there. Go explore.


---

### Answer by username_3 on 2017-11-28. Score: 7

There are numerous directions to go which start moving you beyond ordinary least squares (OLS), linear regression. The universe of statistical methods is large!

Two books that I particularly enjoyed are *Econometrics* by Hayashi and *Elements of Statistical Learning* by Hastie et. al. Looking back at your question, these books may be too advanced. But maybe not. An easier version of the latter is *An Introduction to Statistical Learning* (and may be interesting from the perspective of a broader exposure to data science than just username_0ometrics).

* Hayashi's *Econometrics* introduces a variety of methods through the lens of GMM and with an eye towards time-series username_0ometrics.
* *Elements of Statistical Learning* is a modern classic of the statistics, machine learning literature. It's great for opening your eyes to methods outside of traditional username_0ometrics.

### Some examples beyond ordinary least squares...

* [Maximum likelihood estimation (MLE)](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation)

  You must know this if you continue doing statistics. It's a ubiquitous workhorse.

  If you can specify the [likelihood function](https://en.wikipedia.org/wiki/Likelihood_function) then parameters of the likelihood function can be estimated by maximizing the likelihood function. In certain special cases, (eg. linear regression with conditionally normal error terms) the OLS estimator is the MLE estimator. You've undoubtedly encountered MLE estimation before if you've estimated a [logit model](https://en.wikipedia.org/wiki/Logistic_regression). MLE is all over physics, engineering, and the sciences.

  There are issues with applying MLE to username_0omic models though. Often we know that an overall username_0omic model is false. A model will imply certain facts that are entirely fallacious. Forcing the model to match the data in a maximum likelihood sense may not induce a useful choice of parameters. GMM can be used as an alternative approach to selectively test certain predictions of a model.
* [GMM](https://en.wikipedia.org/wiki/Generalized_method_of_moments) is another broad method for estimating parameters based upon moment conditions that in expectation should be zero. Hayashi's book *Econometrics* develops ordinary least squares regression, instrumental variables, maximum likelihood, and other methods as special cases of GMM with different moment conditions. OLS can be thought of as GMM using the orthogonality condition of the regressors and the error terms. MLE can be derived as GMM on the [score](https://en.wikipedia.org/wiki/Score_(statistics)).

  [A John Cochrane ode to GMM is here.](https://johnhcochrane.blogspot.com/2013/10/hansens-nobel.html)
* [Matching methods](https://en.wikipedia.org/wiki/Matching_(statistics)) for estimating causal effects are common in certain areas of username_0omics.

  The idea is to match a treated entity with an untreated entity based upon observable characteristics. A widely used technique for example is [propensity score matching](https://en.wikipedia.org/wiki/Propensity_score_matching)
* There are all kinds of variations on classic linear methods:

  The idea here is to start with ordinary least squares but then to bias coefficient estimates towards zero to reduce overfitting and improve out of sample prediction.

  + [Ridge regression](https://en.wikipedia.org/wiki/Tikhonov_regularization)
  + [Lasso](https://en.wikipedia.org/wiki/Lasso_(statistics))

### References

[Fumio, Hayashi, 2000, *Econometrics*](https://rads.stackoverflow.com/amzn/click/0691010188)

[Hastie, Trevor, Robert Tibshirani, Jerome Friedman, 2009, *Elements of Statistical Learning*](https://rads.stackoverflow.com/amzn/click/B00475AS2E)

[James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2017, *An Introduction to Statistical Learning*](https://rads.stackoverflow.com/amzn/click/1461471370)

> **Comments:**
> **username_0**: Thanks! I read Introduction to statistical learning and mainly motivation for this question came from disappointment of not seeing any regularization methods used in our username_0omics and username_0ometrics classes...thanks :)  
> **username_3**: @username_0 You're going to see more machine learning techniques used in username_0omics, especially for problems or subproblems where the interest is in a forecast $\hat{y}$ and you don't care as much why or how you got that forecast. Be aware though that many machine learning techniques can be problematic for $\hat{\beta}$ problems where you're trying to estimate a specific parameter (eg. what's the causal effect of $x$). Machine learning techniques etc... may also provoke hostile questions from many username_0omists who don't know the techniques and/or view them with suspicion.  

---

### Answer by username_1 on 2017-11-25. Score: 4

Linear regression, despite its simplicity, it is actually a very powerful tool. That's why it's everywhere in username_0ometrics, to give you an example you're maybe familiar with, consider an [auto-regressive](https://en.wikipedia.org/wiki/Autoregressive_model) model, turns out you can write the future state of a variable that follows this model as a linear combination of previous states

$$
X\_t = C + \phi\_1 X\_{t-1} + \phi\_2 X\_{t-2} + \cdots + \phi\_p X\_{t-p} + \eta\_t \tag{1}
$$

so, if you know the weights $\{\phi\_k \}\_{k=1}^p$ you can predict what's going to be the future values of $X$. The interesting part is that these numbers are obtained through linear regression: just call $y = X\_t$, $x\_1 = X\_{t-1}, x\_2 = X\_{t-2},\cdots$ and realize Eq. (1) can be written as

$$
y = C + \phi\_1 x\_1 + \phi\_2 x\_2 + \cdots + \phi\_p x\_p + \epsilon \tag{2}
$$

In this sense linear regression can be used for forecasting. But there are other tools, I will link you to [this](https://stats.stackexchange.com/questions/10162/how-to-apply-neural-network-to-time-series-forecasting) other thread where you can get an idea on how *neuronal networks* can be used for this task: forecasting time series. But forecasting can also be done with a plethora of methods: [support vector machines](http://www.svms.org/finance/) are popular choices.


---

### Answer by username_4 on 2017-11-27. Score: 2

You can always think of a linear regression model as a first-order [Taylor approximation](https://en.wikipedia.org/wiki/Taylor_series#Approximation_error_and_convergence) of some nonlinear regression model. So linearity itself is not a particularly serious issue.

That being said, there are a couple of common nonlinear regression models that are accessible at the undergraduate level, namely, models with binary dependent variables: [logit and probit](https://liberalarts.utexas.edu/prc/_files/cs/Fall2013_Moore_Logistic_Probit_Regression.pdf).


---

