  Linear regression in $\ell_p$-norm is a canonical optimization problem that
arises in several applications, including sparse recovery, semi-supervised
learning, and signal processing. Generic convex optimization algorithms for
solving $\ell_p$-regression are slow in practice. Iteratively Reweighted Least
Squares (IRLS) is an easy to implement family of algorithms for solving these
problems that has been studied for over 50 years. However, these algorithms
often diverge for p > 3, and since the work of Osborne (1985), it has been an
open problem whether there is an IRLS algorithm that is guaranteed to converge
rapidly for p > 3. We propose p-IRLS, the first IRLS algorithm that provably
converges geometrically for any $p \in [2,\infty).$ Our algorithm is simple to
implement and is guaranteed to find a $(1+\varepsilon)$-approximate solution in
$O(p^{3.5} m^{\frac{p-2}{2(p-1)}} \log \frac{m}{\varepsilon}) \le O_p(\sqrt{m}
\log \frac{m}{\varepsilon} )$ iterations. Our experiments demonstrate that it
performs even better than our theoretical bounds, beats the standard Matlab/CVX
implementation for solving these problems by 10--50x, and is the fastest among
available implementations in the high-accuracy regime.
