  In this paper, we develop upon the emerging topic of loss function learning,
which aims to learn loss functions that significantly improve the performance
of the models trained under them. Specifically, we propose a new meta-learning
framework for learning model-agnostic loss functions via a hybrid
neuro-symbolic search approach. The framework first uses evolution-based
methods to search the space of primitive mathematical operations to find a set
of symbolic loss functions. Second, the set of learned loss functions are
subsequently parameterized and optimized via an end-to-end gradient-based
training procedure. The versatility of the proposed framework is empirically
validated on a diverse set of supervised learning tasks. Results show that the
meta-learned loss functions discovered by the newly proposed method outperform
both the cross-entropy loss and state-of-the-art loss function learning methods
on a diverse range of neural network architectures and datasets.
