  Pansharpening aims to fuse a registered high-resolution panchromatic image
(PAN) with a low-resolution hyperspectral image (LR-HSI) to generate an
enhanced HSI with high spectral and spatial resolution. Existing pansharpening
approaches neglect using an attention mechanism to transfer HR texture features
from PAN to LR-HSI features, resulting in spatial and spectral distortions. In
this paper, we present a novel attention mechanism for pansharpening called
HyperTransformer, in which features of LR-HSI and PAN are formulated as queries
and keys in a transformer, respectively. HyperTransformer consists of three
main modules, namely two separate feature extractors for PAN and HSI, a
multi-head feature soft attention module, and a spatial-spectral feature fusion
module. Such a network improves both spatial and spectral quality measures of
the pansharpened HSI by learning cross-feature space dependencies and
long-range details of PAN and LR-HSI. Furthermore, HyperTransformer can be
utilized across multiple spatial scales at the backbone for obtaining improved
performance. Extensive experiments conducted on three widely used datasets
demonstrate that HyperTransformer achieves significant improvement over the
state-of-the-art methods on both spatial and spectral quality measures.
Implementation code and pre-trained weights can be accessed at
https://github.com/wgcban/HyperTransformer.
