  Unsupervised domain adaptive (UDA) image segmentation has recently gained
increasing attention, aiming to improve the generalization capability for
transferring knowledge from the source domain to the target domain. However, in
high spatial resolution remote sensing image (RSI), the same category from
different domains (\emph{e.g.}, urban and rural) can appear to be totally
different with extremely inconsistent distributions, which heavily limits the
UDA accuracy. To address this problem, in this paper, we propose a novel Deep
Covariance Alignment (DCA) model for UDA RSI segmentation. The DCA can
explicitly align category features to learn shared domain-invariant
discriminative feature representations, which enhances the ability of model
generalization. Specifically, a Category Feature Pooling (CFP) module is first
employed to extract category features by combining the coarse outputs and the
deep features. Then, we leverage a novel Covariance Regularization (CR) to
enforce the intra-category features to be closer and the inter-category
features to be further separate. Compared with the existing category alignment
methods, our CR aims to regularize the correlation between different dimensions
of the features and thus performs more robustly when dealing with the divergent
category features of imbalanced and inconsistent distributions. Finally, we
propose a stagewise procedure to train the DCA in order to alleviate the error
accumulation. Experiments on both Rural-to-Urban and Urban-to-Rural scenarios
of the LoveDA dataset demonstrate the superiority of our proposed DCA over
other state-of-the-art UDA segmentation methods. Code is available at
https://github.com/Luffy03/DCA.
