  Previous work in Neural Loss Function Search (NLFS) has shown a lack of
correlation between smaller surrogate functions and large convolutional neural
networks with massive regularization. We expand upon this research by revealing
another disparity that exists, correlation between different types of image
augmentation techniques. We show that different loss functions can perform well
on certain image augmentation techniques, while performing poorly on others. We
exploit this disparity by performing an evolutionary search on five types of
image augmentation techniques in the hopes of finding image augmentation
specific loss functions. The best loss functions from each evolution were then
taken and transferred to WideResNet-28-10 on CIFAR-10 and CIFAR-100 across each
of the five image augmentation techniques. The best from that were then taken
and evaluated by fine-tuning EfficientNetV2Small on the CARS, Oxford-Flowers,
and Caltech datasets across each of the five image augmentation techniques.
Multiple loss functions were found that outperformed cross-entropy across
multiple experiments. In the end, we found a single loss function, which we
called the inverse bessel logarithm loss, that was able to outperform
cross-entropy across the majority of experiments.
