  Neural networks are trained by minimizing a loss function that defines the
discrepancy between the predicted model output and the target value. The
selection of the loss function is crucial to achieve task-specific behaviour
and highly influences the capability of the model. A variety of loss functions
have been proposed for a wide range of tasks affecting training and model
performance. For classification tasks, the cross entropy is the de-facto
standard and usually the first choice. Here, we try to experimentally challenge
the well-known loss functions, including cross entropy (CE) loss, by utilizing
the genetic programming (GP) approach, a population-based evolutionary
algorithm. GP constructs loss functions from a set of operators and leaf nodes
and these functions are repeatedly recombined and mutated to find an optimal
structure. Experiments were carried out on different small-sized datasets
CIFAR-10, CIFAR-100 and Fashion-MNIST using an Inception model. The 5 best
functions found were evaluated for different model architectures on a set of
standard datasets ranging from 2 to 102 classes and very different sizes. One
function, denoted as Next Generation Loss (NGL), clearly stood out showing same
or better performance for all tested datasets compared to CE. To evaluate the
NGL function on a large-scale dataset, we tested its performance on the
Imagenet-1k dataset where it showed improved top-1 accuracy compared to models
trained with identical settings and other losses. Finally, the NGL was trained
on a segmentation downstream task for Pascal VOC 2012 and COCO-Stuff164k
datasets improving the underlying model performance.
