  Regularization aims to improve prediction performance of a given statistical
modeling approach by moving to a second approach which achieves worse training
error but is expected to have fewer degrees of freedom, i.e., better agreement
between training and prediction error. We show here, however, that this
expected behavior does not hold in general. In fact, counter examples are given
that show regularization can increase the degrees of freedom in simple
situations, including lasso and ridge regression, which are the most common
regularization approaches in use. In such situations, the regularization
increases both training error and degrees of freedom, and is thus inherently
without merit. On the other hand, two important regularization scenarios are
described where the expected reduction in degrees of freedom is indeed
guaranteed: (a) all symmetric linear smoothers, and (b) linear regression
versus convex constrained linear regression (as in the constrained variant of
ridge regression and lasso).
