  Recent works have demonstrated that global covariance pooling (GCP) has the
ability to improve performance of deep convolutional neural networks (CNNs) on
visual classification task. Despite considerable advance, the reasons on
effectiveness of GCP on deep CNNs have not been well studied. In this paper, we
make an attempt to understand what deep CNNs benefit from GCP in a viewpoint of
optimization. Specifically, we explore the effect of GCP on deep CNNs in terms
of the Lipschitzness of optimization loss and the predictiveness of gradients,
and show that GCP can make the optimization landscape more smooth and the
gradients more predictive. Furthermore, we discuss the connection between GCP
and second-order optimization for deep CNNs. More importantly, above findings
can account for several merits of covariance pooling for training deep CNNs
that have not been recognized previously or fully explored, including
significant acceleration of network convergence (i.e., the networks trained
with GCP can support rapid decay of learning rates, achieving favorable
performance while significantly reducing number of training epochs), stronger
robustness to distorted examples generated by image corruptions and
perturbations, and good generalization ability to different vision tasks, e.g.,
object detection and instance segmentation. We conduct extensive experiments
using various deep CNN models on diversified tasks, and the results provide
strong support to our findings.
