  Pan-sharpening, as one of the most commonly used techniques in remote sensing
systems, aims to inject spatial details from panchromatic images into
multispectral images (MS) to obtain high-resolution multispectral images. Since
deep learning has received widespread attention because of its powerful fitting
ability and efficient feature extraction, a variety of pan-sharpening methods
have been proposed to achieve remarkable performance. However, current
pan-sharpening methods usually require the paired panchromatic (PAN) and MS
images as input, which limits their usage in some scenarios. To address this
issue, in this paper we observe that the spatial details from PAN images are
mainly high-frequency cues, i.e., the edges reflect the contour of input PAN
images. This motivates us to develop a PAN-agnostic representation to store
some base edges, so as to compose the contour for the corresponding PAN image
via them. As a result, we can perform the pan-sharpening task with only the MS
image when inference. To this end, a memory-based network is adapted to extract
and memorize the spatial details during the training phase and is used to
replace the process of obtaining spatial information from PAN images when
inference, which is called Memory-based Spatial Details Network (MSDN).
Finally, we integrate the proposed MSDN module into the existing deep
learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening
network. With extensive experiments on the Gaofen1 and WorldView-4 satellites,
we verify that our method constructs good spatial details without PAN images
and achieves the best performance. The code is available at
https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git.
