  All machine learning algorithms use a loss, cost, utility or reward function
to encode the learning objective and oversee the learning process. This
function that supervises learning is a frequently unrecognized hyperparameter
that determines how incorrect outputs are penalized and can be tuned to improve
performance. This paper shows that training speed and final accuracy of neural
networks can significantly depend on the loss function used to train neural
networks. In particular derivative values can be significantly different with
different loss functions leading to significantly different performance after
gradient descent based Backpropagation (BP) training. This paper explores the
effect on performance of using new loss functions that are also convex but
penalize errors differently compared to the popular Cross-entropy loss. Two new
classification loss functions that significantly improve performance on a wide
variety of benchmark tasks are proposed. A new loss function call smooth
absolute error that outperforms the Squared error, Huber and Log-Cosh losses on
datasets with significantly many outliers is proposed. This smooth absolute
error loss function is infinitely differentiable and more closely approximates
the absolute error loss compared to the Huber and Log-Cosh losses used for
robust regression.
