I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear model.

To paraphrase Shalizi's example: suppose you have a model $Y = aX + \epsilon$, where $a$ is known. Then $\newcommand{\Var}{\mathrm{Var}}\Var[Y] = a^2 \Var[x] + \Var[\epsilon]$ and the amount of explained variance is $a^2 \Var[X]$, so $R^2 = \frac{a^2 \Var[x]}{a^2 \Var[X] + \Var[\epsilon]}$. This goes to 0 as $\Var[X] \rightarrow 0$ and to 1 as $\Var[X] \rightarrow \infty$.

Conversely, you can get high $R^2$ even when your model is noticeably non-linear. (Anyone have a good example offhand?)

So when is $R^2$ a useful statistic, and when should it be ignored?

To address the first question, consider the model

$$Y = X + \sin(X) + \varepsilon$$

with iid $\varepsilon$ of mean zero and finite variance.  As the range of $X$ (thought of as fixed or random) increases, $R^2$ goes to 1.  Nevertheless, if the variance of $\varepsilon$ is small (around 1 or less), the data are "noticeably non-linear."  In the plots, $var(\varepsilon)=1$.





Incidentally, an easy way to get a small $R^2$ is to slice the independent variables into narrow ranges.  The regression (using exactly the same model) within each range will have a low $R^2$ even when the full regression based on all the data has a high $R^2$.  Contemplating this situation is an informative exercise and good preparation for the second question.

Both the following plots use the same data.  The $R^2$ for the full regression is 0.86.  The $R^2$ for the slices (of width 1/2 from -5/2 to 5/2) are .16, .18, .07, .14, .08, .17, .20, .12, .01, .00, reading left to right.  If anything, the fits get better in the sliced situation because the 10 separate lines can more closely conform to the data within their narrow ranges.  Although the $R^2$ for all the slices are far below the full $R^2$, neither the strength of the relationship, the linearity, nor indeed any aspect of the data (except the range of $X$ used for the regression) has changed.





(One might object that this slicing procedure changes the distribution of $X$.  That is true, but it nevertheless corresponds with the most common use of $R^2$ in fixed-effects modeling and reveals the degree to which $R^2$ is telling us about the variance of $X$ in the random-effects situation.  In particular, when $X$ is constrained to vary within a smaller interval of its natural range, $R^2$ will usually drop.)

The basic problem with $R^2$ is that it depends on too many things (even when adjusted in multiple regression), but most especially on the variance of the independent variables and the variance of the residuals.  Normally it tells us nothing about "linearity" or "strength of relationship" or even "goodness of fit" for comparing a sequence of models. 

Most of the time you can find a better statistic than $R^2$.  For model selection you can look to AIC and BIC; for expressing the adequacy of a model, look at the variance of the residuals.  

This brings us finally to the second question.  One situation in which $R^2$ might have some use is when the independent variables are set to standard values, essentially controlling for the effect of their variance.  Then $1 - R^2$ is really a proxy for the variance of the residuals, suitably standardized.
