  Current online learning methods suffer issues such as lower convergence rates
and limited capability to select important features compared to their offline
counterparts. In this paper, a novel framework for online learning based on
running averages is proposed. Many popular offline regularized methods such as
Lasso, Elastic Net, Minimax Concave Penalty (MCP), and Feature Selection with
Annealing (FSA) have their online versions introduced in this framework. The
equivalence between the proposed online methods and their offline counterparts
is proved, and then novel theoretical true support recovery and convergence
guarantees are provided for some of the methods in this framework. Numerical
experiments indicate that the proposed methods enjoy high true support recovery
accuracy and a faster convergence rate compared with conventional online and
offline algorithms. Finally, applications to large datasets are presented,
where again the proposed framework shows competitive results compared to
popular online and offline algorithms.
