  Stochastic optimization methods encounter new challenges in the realm of
streaming, characterized by a continuous flow of large, high-dimensional data.
While first-order methods, like stochastic gradient descent, are the natural
choice, they often struggle with ill-conditioned problems. In contrast,
second-order methods, such as Newton's methods, offer a potential solution, but
their computational demands render them impractical. This paper introduces
adaptive stochastic optimization methods that bridge the gap between addressing
ill-conditioned problems while functioning in a streaming context. Notably, we
present an adaptive inversion-free Newton's method with a computational
complexity matching that of first-order methods, $\mathcal{O}(dN)$, where $d$
represents the number of dimensions/features, and $N$ the number of data.
Theoretical analysis confirms their asymptotic efficiency, and empirical
evidence demonstrates their effectiveness, especially in scenarios involving
complex covariance structures and challenging initializations. In particular,
our adaptive Newton's methods outperform existing methods, while maintaining
favorable computational efficiency.
