  In this paper, we develop upon the topic of loss function learning, an
emergent meta-learning paradigm that aims to learn loss functions that
significantly improve the performance of the models trained under them.
Specifically, we propose a new meta-learning framework for task and
model-agnostic loss function learning via a hybrid search approach. The
framework first uses genetic programming to find a set of symbolic loss
functions. Second, the set of learned loss functions is subsequently
parameterized and optimized via unrolled differentiation. The versatility and
performance of the proposed framework are empirically validated on a diverse
set of supervised learning tasks. Results show that the learned loss functions
bring improved convergence, sample efficiency, and inference performance on
tabulated, computer vision, and natural language processing problems, using a
variety of task-specific neural network architectures.
