  Pansharpening is to fuse a multispectral image (MSI) of
low-spatial-resolution (LR) but rich spectral characteristics with a
panchromatic image (PAN) of high-spatial-resolution (HR) but poor spectral
characteristics. Traditional methods usually inject the extracted
high-frequency details from PAN into the up-sampled MSI. Recent deep learning
endeavors are mostly supervised assuming the HR MSI is available, which is
unrealistic especially for satellite images. Nonetheless, these methods could
not fully exploit the rich spectral characteristics in the MSI. Due to the wide
existence of mixed pixels in satellite images where each pixel tends to cover
more than one constituent material, pansharpening at the subpixel level becomes
essential. In this paper, we propose an unsupervised pansharpening (UP) method
in a deep-learning framework to address the above challenges based on the
self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this
paper is three-fold. First, the self-attention mechanism is proposed where the
spatial varying detail extraction and injection functions are estimated
according to the attention representations indicating spectral characteristics
of the MSI with sub-pixel accuracy. Second, such attention representations are
derived from mixed pixels with the proposed stacked attention network powered
with a stick-breaking structure to meet the physical constraints of mixed pixel
formulations. Third, the detail extraction and injection functions are spatial
varying based on the attention representations, which largely improves the
reconstruction accuracy. Extensive experimental results demonstrate that the
proposed approach is able to reconstruct sharper MSI of different types, with
more details and less spectral distortion as compared to the state-of-the-art.
