  This work formalizes and analyzes Gaussian smoothing applied to three
optimization methods, namely: stochastic gradient descent (GSmoothSGD); Adam
(GSmoothAdam) and; stochastic variance reduced gradient (GSmoothSVRG), in the
context of training deep learning models. Gaussian smoothing can enhance the
behavior of functions by flattening out small fluctuations, thereby reducing
the likelihood of gradient descent algorithms converging to suboptimal local
minima. The resulting new approaches aim to reduce the complexity of the loss
landscape while improving robustness to noise and generalization, which
enhances the effectiveness of the base algorithms and facilitating their
convergence towards global minima. In addition, the vast majority of efforts
aimed at applying Gaussian smoothing to a neural network involve zero-order
approximations, resulting in decreased computational efficiency by increasing
the time required to train compared to automatic differentiation. To overcome
this computational burden, we explicitly construct new smoothed neural networks
by deriving Gaussian smoothed versions of the loss functions coming from both
feedforward and convolutional network architectures. Finally, we present
several numerical examples that exemplify our theoretical results and
demonstrate the improved performance of the proposed smoothing algorithms
compared to their unsmoothed counterparts.
