  Bilevel optimization has been widely used in many machine learning
applications such as hyperparameter optimization and meta learning. Recently,
many simple stochastic gradient descent(SGD) type algorithms(without using
momentum and variance techniques) have been proposed to solve the bilevel
optimization problems. However, all the existing simple SGD type algorithms
estimate the hypergradient via stochastic estimation of Neumann series. In the
paper, we propose to estimate the hypergradient via SGD-based Estimation(i.e.,
solving the linear system with SGD). By using warm start initialization
strategy, a new simple SGD type algorithm SSGD based on SGD-based Estimation is
proposed. We provide the convergence rate guarantee for SSGD and show that SSGD
outperforms the best known computational complexity achieved by the existing
simple SGD type algorithms. Our experiments validate our theoretical results
and demonstrate the efficiency of our proposed algorithm SSGD in hyperparameter
optimization applications.
