  Global covariance pooling (GCP) aims at exploiting the second-order
statistics of the convolutional feature. Its effectiveness has been
demonstrated in boosting the classification performance of Convolutional Neural
Networks (CNNs). Singular Value Decomposition (SVD) is used in GCP to compute
the matrix square root. However, the approximate matrix square root calculated
using Newton-Schulz iteration \cite{li2018towards} outperforms the accurate one
computed via SVD \cite{li2017second}. We empirically analyze the reason behind
the performance gap from the perspectives of data precision and gradient
smoothness. Various remedies for computing smooth SVD gradients are
investigated. Based on our observation and analyses, a hybrid training protocol
is proposed for SVD-based GCP meta-layers such that competitive performances
can be achieved against Newton-Schulz iteration. Moreover, we propose a new GCP
meta-layer that uses SVD in the forward pass, and Pad\'e Approximants in the
backward propagation to compute the gradients. The proposed meta-layer has been
integrated into different CNN models and achieves state-of-the-art performances
on both large-scale and fine-grained datasets.
