  In this paper, we propose a new Hessian inverse free Fully Single Loop
Algorithm (FSLA) for bilevel optimization problems. Classic algorithms for
bilevel optimization admit a double loop structure which is computationally
expensive. Recently, several single loop algorithms have been proposed with
optimizing the inner and outer variable alternatively. However, these
algorithms not yet achieve fully single loop. As they overlook the loop needed
to evaluate the hyper-gradient for a given inner and outer state. In order to
develop a fully single loop algorithm, we first study the structure of the
hyper-gradient and identify a general approximation formulation of
hyper-gradient computation that encompasses several previous common approaches,
e.g. back-propagation through time, conjugate gradient, \emph{etc.} Based on
this formulation, we introduce a new state variable to maintain the historical
hyper-gradient information. Combining our new formulation with the alternative
update of the inner and outer variables, we propose an efficient fully single
loop algorithm. We theoretically show that the error generated by the new state
can be bounded and our algorithm converges with the rate of $O(\epsilon^{-2})$.
Finally, we verify the efficacy our algorithm empirically through multiple
bilevel optimization based machine learning tasks.
