I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out.

I have a least squares regression model, with one independent variable and one dependent variable. The relationship is not significant. Now I add a second independent variable. Now the relationship between the first independent variable and the dependent variable becomes significant.

How does this work? This is probably demonstrating some issue with my understanding, but to me, but I do not see how adding this second independent variable can make the first significant.

Although collinearity (of predictor variables) is a possible explanation, I would like to suggest it is not an illuminating explanation because we know collinearity is related to "common information" among the predictors, so there is nothing mysterious or counter-intuitive about the side effect of introducing a second correlated predictor into the model.

Let us then consider the case of two predictors that are truly orthogonal: there is absolutely no collinearity among them.  A remarkable change in significance can still happen.

Designate the predictor variables $X_1$ and $X_2$ and let $Y$ name the predictand.  The regression of $Y$ against $X_1$ will fail to be significant when the variation in $Y$ around its mean is not appreciably reduced when $X_1$ is used as the independent variable.  When that variation is strongly associated with a second variable $X_2$, however, the situation changes.  Recall that multiple regression of $Y$ against $X_1$ and $X_2$ is equivalent to


Separately regress $Y$ and $X_1$ against $X_2$.
Regress the $Y$ residuals against the $X_1$ residuals.


The residuals from the first step have removed the effect of $X_2$.  When $X_2$ is closely correlated with $Y$, this can expose a relatively small amount of variation that had previously been masked.  If this variation is associated with $X_1$, we obtain a significant result.



All this might perhaps be clarified with a concrete example.  To begin, let's use R to generate two orthogonal independent variables along with some independent random error $\varepsilon$:

n <- 32
set.seed(182)
u <-matrix(rnorm(2*n), ncol=2)
u0 <- cbind(u[,1] - mean(u[,1]), u[,2] - mean(u[,2]))
x <- svd(u0)$u
eps <- rnorm(n)


(The svd step assures the two columns of matrix x (representing $X_1$ and $X_2$) are orthogonal, ruling out collinearity as a possible explanation of any subsequent results.)

Next, create $Y$ as a linear combination of the $X$'s and the error.  I have adjusted the coefficients to produce the counter-intuitive behavior:

y <-  x %*% c(0.05, 1) + eps * 0.01


This is a realization of the model $Y \sim_{iid} N(0.05 X_1 + 1.00 X_2, 0.01^2)$ with $n=32$ cases.

Look at the two regressions in question.  First, regress $Y$ against $X_1$ only:

> summary(lm(y ~ x[,1]))
...
             Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.002576   0.032423  -0.079    0.937
x[, 1]       0.068950   0.183410   0.376    0.710


The high p-value of 0.710 shows that $X_1$ is completely non-significant.

Next, regress $Y$ against $X_1$ and $X_2$:

> summary(lm(y ~ x))
...
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.002576   0.001678  -1.535    0.136    
x1           0.068950   0.009490   7.265 5.32e-08 ***
x2           1.003276   0.009490 105.718  < 2e-16 ***


Suddenly, in the presence of $X_2$, $X_1$ is strongly significant, as indicated by the near-zero p-values for both variables.

We can visualize this behavior by means of a scatterplot matrix of the variables $X_1$, $X_2$, and $Y$ along with the residuals used in the two-step characterization of multiple regression above.  Because $X_1$ and $X_2$ are orthogonal, the $X_1$ residuals will be the same as $X_1$ and therefore need not be redrawn.  We will include the residuals of $Y$ against $X_2$ in the scatterplot matrix, giving this figure:

lmy <- lm(y ~ x[,2])
d <- data.frame(X1=x[,1], X2=x[,2], Y=y, RY=residuals(lmy))
plot(d)


Here is a rendering of it (with a little prettification):



This matrix of graphics has four rows and four columns, which I will count down from the top and from left to right.

Notice:


The $(X_1, X_2)$ scatterplot in the second row and first column confirms the orthogonality of these predictors: the least squares line is horizontal and correlation is zero.
The $(X_1, Y)$ scatterplot in the third row and first column exhibits the slight but completely insignificant relationship reported by the first regression of $Y$ against $X_1$.  (The correlation coefficient, $\rho$, is only $0.07$).
The $(X_2, Y)$ scatterplot in the third row and second column shows the strong relationship between $Y$ and the second independent variable.  (The correlation coefficient is $0.996$).
The fourth row examines the relationships between the residuals of $Y$ (regressed against $X_2$) and other variables:


The vertical scale shows that the residuals are (relatively) quite small: we couldn't easily see them in the scatterplot of $Y$ against $X_2$.
The residuals are strongly correlated with $X_1$ ($\rho = 0.80$).  The regression against $X_2$ has unmasked this previously hidden behavior.
By construction, there is no remaining correlation between the residuals and $X_2$.
There is little correlation between $Y$ and these residuals ($\rho = 0.09$).  This shows how the residuals can behave entirely differently than $Y$ itself.  That's how $X_1$ can suddenly be revealed as a significant contributor to the regression.



Finally, it is worth remarking that the two estimates of the $X_1$ coefficient (both equal to $0.06895$, not far from the intended value of $0.05$) agree only because $X_1$ and $X_2$ are orthogonal.  Except in designed experiments, it is rare for orthogonality to hold exactly.  A departure from orthogonality usually causes the coefficient estimates to change.
