  Learned optimizers are a crucial component of meta-learning. Recent
advancements in scalable learned optimizers have demonstrated their superior
performance over hand-designed optimizers in various tasks. However, certain
characteristics of these models, such as an unstable learning curve, limited
ability to handle unseen tasks and network architectures, difficult-to-control
behaviours, and poor performance in fine-tuning tasks impede their widespread
adoption. To tackle the issue of generalization in scalable learned optimizers,
we propose a hybrid-update-based (HUB) optimization strategy inspired by recent
advancements in hard prompt tuning and result selection techniques used in
large language and vision models. This approach can be easily applied to any
task that involves hand-designed or learned optimizer. By incorporating
hand-designed optimizers as the second component in our hybrid approach, we are
able to retain the benefits of learned optimizers while stabilizing the
training process and, more importantly, improving testing performance. We
validate our design through a total of 17 tasks, consisting of thirteen
training from scratch and four fine-tuning settings. These tasks vary in model
sizes, architectures, or dataset sizes, and the competing optimizers are
hyperparameter-tuned. We outperform all competitors in 94% of the tasks with
better testing performance. Furthermore, we conduct a theoretical analysis to
examine the potential impact of our hybrid strategy on the behaviours and
inherited traits of learned optimizers.
