  Stochastic gradients closely relate to both optimization and generalization
of deep neural networks (DNNs). Some works attempted to explain the success of
stochastic optimization for deep learning by the arguably heavy-tail properties
of gradient noise, while other works presented theoretical and empirical
evidence against the heavy-tail hypothesis on gradient noise. Unfortunately,
formal statistical tests for analyzing the structure and heavy tails of
stochastic gradients in deep learning are still under-explored. In this paper,
we mainly make two contributions. First, we conduct formal statistical tests on
the distribution of stochastic gradients and gradient noise across both
parameters and iterations. Our statistical tests reveal that dimension-wise
gradients usually exhibit power-law heavy tails, while iteration-wise gradients
and stochastic gradient noise caused by minibatch training usually do not
exhibit power-law heavy tails. Second, we further discover that the covariance
spectra of stochastic gradients have the power-law structures overlooked by
previous studies and present its theoretical implications for training of DNNs.
While previous studies believed that the anisotropic structure of stochastic
gradients matters to deep learning, they did not expect the gradient covariance
can have such an elegant mathematical structure. Our work challenges the
existing belief and provides novel insights on the structure of stochastic
gradients in deep learning.
