  Huber's criterion can be used for robust joint estimation of regression and
scale parameters in the linear model. Huber's (Huber, 1981) motivation for
introducing the criterion stemmed from non-convexity of the joint maximum
likelihood objective function as well as non-robustness (unbounded influence
function) of the associated ML-estimate of scale. In this paper, we illustrate
how the original algorithm proposed by Huber can be set within the block-wise
minimization majorization framework. In addition, we propose novel
data-adaptive step sizes for both the location and scale, which are further
improving the convergence. We then illustrate how Huber's criterion can be used
for sparse learning of underdetermined linear model using the iterative hard
thresholding approach. We illustrate the usefulness of the algorithms in an
image denoising application and simulation studies.
