  Fusion of a panchromatic (PAN) image and corresponding multispectral (MS)
image is also known as pansharpening, which aims to combine abundant spatial
details of PAN and spectral information of MS. Due to the absence of
high-resolution MS images, available deep-learning-based methods usually follow
the paradigm of training at reduced resolution and testing at both reduced and
full resolution. When taking original MS and PAN images as inputs, they always
obtain sub-optimal results due to the scale variation. In this paper, we
propose to explore the self-supervised representation of pansharpening by
designing a cross-predictive diffusion model, named CrossDiff. It has two-stage
training. In the first stage, we introduce a cross-predictive pretext task to
pre-train the UNet structure based on conditional DDPM, while in the second
stage, the encoders of the UNets are frozen to directly extract spatial and
spectral features from PAN and MS, and only the fusion head is trained to adapt
for pansharpening task. Extensive experiments show the effectiveness and
superiority of the proposed model compared with state-of-the-art supervised and
unsupervised methods. Besides, the cross-sensor experiments also verify the
generalization ability of proposed self-supervised representation learners for
other satellite's datasets. We will release our code for reproducibility.
