  Dominant pan-sharpening frameworks simply concatenate the MS stream and the
PAN stream once at a specific level. This way of fusion neglects the
multi-level spectral-spatial correlation between the two streams, which is
vital to improving the fusion performance. In consideration of this, we propose
a Multi-level and Enhanced Spectral-Spatial Fusion Network (MESSFN) with the
following innovations: First, to fully exploit and strengthen the above
correlation, a Hierarchical Multi-level Fusion Architecture (HMFA) is carefully
designed. A novel Spectral-Spatial (SS) stream is established to hierarchically
derive and fuse the multi-level prior spectral and spatial expertise from the
MS stream and the PAN stream. This helps the SS stream master a joint
spectral-spatial representation in the hierarchical network for better modeling
the fusion relationship. Second, to provide superior expertise, consequently,
based on the intrinsic characteristics of the MS image and the PAN image, two
feature extraction blocks are specially developed. In the MS stream, a Residual
Spectral Attention Block (RSAB) is proposed to mine the potential spectral
correlations between different spectra of the MS image through adjacent
cross-spectrum interaction. While in the PAN stream, a Residual Multi-scale
Spatial Attention Block (RMSAB) is proposed to capture multi-scale information
and reconstruct precise high-frequency details from the PAN image through an
improved spatial attention-based inception structure. The spectral and spatial
feature representations are enhanced. Extensive experiments on two datasets
demonstrate that the proposed network is competitive with or better than
state-of-the-art methods. Our code can be found in github.
