  This research proposes to use the Moreau-Yosida envelope to stabilize the
convergence behavior of bi-level Hyperparameter optimization solvers, and
introduces the new algorithm called Moreau-Yosida regularized Hyperparameter
Optimization (MY-HPO) algorithm. Theoretical analysis on the correctness of the
MY-HPO solution and initial convergence analysis is also provided. Our
empirical results show significant improvement in loss values for a fixed
computation budget, compared to the state-of-art bi-level HPO solvers.
