  The classical iteratively reweighted least-squares (IRLS) algorithm aims to
recover an unknown signal from linear measurements by performing a sequence of
weighted least squares problems, where the weights are recursively updated at
each step. Varieties of this algorithm have been shown to achieve favorable
empirical performance and theoretical guarantees for sparse recovery and
$\ell_p$-norm minimization. Recently, some preliminary connections have also
been made between IRLS and certain types of non-convex linear neural network
architectures that are observed to exploit low-dimensional structure in
high-dimensional linear models. In this work, we provide a unified asymptotic
analysis for a family of algorithms that encompasses IRLS, the recently
proposed lin-RFM algorithm (which was motivated by feature learning in neural
networks), and the alternating minimization algorithm on linear diagonal neural
networks. Our analysis operates in a "batched" setting with i.i.d. Gaussian
covariates and shows that, with appropriately chosen reweighting policy, the
algorithm can achieve favorable performance in only a handful of iterations. We
also extend our results to the case of group-sparse recovery and show that
leveraging this structure in the reweighting scheme provably improves test
error compared to coordinate-wise reweighting.
