  Clipping the gradient is a known approach to improving gradient descent, but
requires hand selection of a clipping threshold hyperparameter. We present
AutoClip, a simple method for automatically and adaptively choosing a gradient
clipping threshold, based on the history of gradient norms observed during
training. Experimental results show that applying AutoClip results in improved
generalization performance for audio source separation networks. Observation of
the training dynamics of a separation network trained with and without AutoClip
show that AutoClip guides optimization into smoother parts of the loss
landscape. AutoClip is very simple to implement and can be integrated readily
into a variety of applications across multiple domains.
