Ridge regression coefficient estimate $\hat{\beta}^R$ are the values that minimize the 

$$ \text{RSS} + \lambda \sum_{j=1}^p\beta_j^2. $$

My questions are:


If $\lambda = 0$, then we see that the expression above reduces to the usual RSS. What if $\lambda \to \infty$?  I do not understand the textbook explanation of the behaviour of the coefficients.
To aid in understanding the concept behind a particular term, why is the term called RIDGE Regression? (Why ridge?) And what could have been wrong with the usual/common regression that there is a need to introduce a new concept called ridge regression? 


Your insights would be great.  

Since you ask for insights, I'm going to take a fairly intuitive approach rather than a more mathematical tack:


Following the concepts in my answer here, we can formulate a ridge regression as a regression with dummy data by adding $p$ (in your formulation) observations, where $y_{n+j}=0$, $x_{j,n+j}=\sqrt{\lambda}$ and $x_{i,n+j}=0$ for $i\neq j$. If you write out the new RSS for this expanded data set, you'll see the additional observations each add a term of the form $(0-\sqrt{\lambda}\beta_j)^2=\lambda\beta_j^2$, so the new RSS is the original $\text{RSS} + \lambda \sum_{j=1}^p\beta_j^2$ -- and minimizing the RSS on this new, expanded data set is the same as minimizing the ridge regression criterion.

So what can we see here? As $\lambda$ increases, the additional $x$-rows  each have one component that increases, and so the influence of these points also increases. They pull the fitted hyperplane toward themselves. Then as $\lambda$ and the corresponding components of the $x$'s go off to infinity, all the involved coefficients "flatten out" to $0$.

That is, as $\lambda\to\infty$, the penalty will dominate the minimization, so the $\beta$s will go to zero. If the intercept is not penalized (the usual case) then the model shrinks more and more toward the mean of the response.
I'll give an intuitive sense of why we're talking about ridges first (which also suggests why it's needed), then tackle a little history. The first is adapted from my answer here:

If there's multicollinearity, you get a "ridge" in the likelihood function (likelihood is a function of the $\beta$'s). This in turn yields a long "valley" in the RSS (since RSS=$-2\log\mathcal{L}$). 

Ridge regression "fixes" the ridge - it adds a penalty that turns the ridge into a nice peak in likelihood space, equivalently a nice depression in the criterion we're minimizing:


[Clearer image]

The actual story behind the name is a little more complicated. In 1959 A.E. Hoerl [1] introduced ridge analysis for response surface methodology, and it very soon [2] became adapted to dealing with multicollinearity in regression ('ridge regression'). See for example, the discussion by R.W. Hoerl in [3], where it  describes Hoerl's (A.E. not R.W.) use of contour plots of the response surface* in the identification of where to head to find local optima (where one 'heads up the ridge'). In ill-conditioned problems, the issue of a very long ridge arises, and insights and methodology from ridge analysis are adapted to the related issue with the likelihood/RSS in regression, producing ridge regression.


* examples of response surface contour plots (in the case of quadratic response) can be seen here (Fig 3.9-3.12).

That is, "ridge" actually refers to the characteristics of the function we were attempting to optimize, rather than to adding a "ridge" (+ve diagonal) to the $X^TX$ matrix (so while ridge regression does add to the diagonal, that's not why we call it 'ridge' regression).

For some additional information on the need for ridge regression, see the first link under list item 2. above.



References:

[1]: Hoerl, A.E. (1959). Optimum solution of many variables equations. Chemical Engineering Progress, 
55 (11) 69-78.

[2]: Hoerl, A.E. (1962). Applications of ridge analysis to regression problems. Chemical Engineering Progress, 
58 (3) 54-59.

[3] Hoerl, R.W. (1985). Ridge Analysis 25 Years Later.
American Statistician, 39 (3), 186-192 
