  Regularization methods allow one to handle a variety of inferential problems
where there are more covariates than cases. This allows one to consider a
potentially enormous number of covariates for a problem. We exploit the power
of these techniques, supersaturating models by augmenting the "natural"
covariates in the problem with an additional indicator for each case in the
data set. We attach a penalty term for these case-specific indicators which is
designed to produce a desired effect. For regression methods with squared error
loss, an $\ell_1$ penalty produces a regression which is robust to outliers and
high leverage cases; for quantile regression methods, an $\ell_2$ penalty
decreases the variance of the fit enough to overcome an increase in bias. The
paradigm thus allows us to robustify procedures which lack robustness and to
increase the efficiency of procedures which are robust. We provide a general
framework for the inclusion of case-specific parameters in regularization
problems, describing the impact on the effective loss for a variety of
regression and classification problems. We outline a computational strategy by
which existing software can be modified to solve the augmented regularization
problem, providing conditions under which such modification will converge to
the optimum solution. We illustrate the benefits of including case-specific
parameters in the context of mean regression and quantile regression through
analysis of NHANES and linguistic data sets.
