  Deep learning based pan-sharpening has received significant research interest
in recent years. Most of existing methods fall into the supervised learning
framework in which they down-sample the multi-spectral (MS) and panchromatic
(PAN) images and regard the original MS images as ground truths to form
training samples. Although impressive performance could be achieved, they have
difficulties generalizing to the original full-scale images due to the scale
gap, which makes them lack of practicability. In this paper, we propose an
unsupervised generative adversarial framework that learns from the full-scale
images without the ground truths to alleviate this problem. We extract the
modality-specific features from the PAN and MS images with a two-stream
generator, perform fusion in the feature domain, and then reconstruct the
pan-sharpened images. Furthermore, we introduce a novel hybrid loss based on
the cycle-consistency and adversarial scheme to improve the performance.
Comparison experiments with the state-of-the-art methods are conducted on
GaoFen-2 and WorldView-3 satellites. Results demonstrate that the proposed
method can greatly improve the pan-sharpening performance on the full-scale
images, which clearly show its practical value. Codes are available at
https://github.com/zhysora/UCGAN.
