  Pan-sharpening involves reconstructing missing high-frequency information in
multi-spectral images with low spatial resolution, using a higher-resolution
panchromatic image as guidance. Although the inborn connection with frequency
domain, existing pan-sharpening research has not almost investigated the
potential solution upon frequency domain. To this end, we propose a novel
Frequency Adaptive Mixture of Experts (FAME) learning framework for
pan-sharpening, which consists of three key components: the Adaptive Frequency
Separation Prediction Module, the Sub-Frequency Learning Expert Module, and the
Expert Mixture Module. In detail, the first leverages the discrete cosine
transform to perform frequency separation by predicting the frequency mask. On
the basis of generated mask, the second with low-frequency MOE and
high-frequency MOE takes account for enabling the effective low-frequency and
high-frequency information reconstruction. Followed by, the final fusion module
dynamically weights high-frequency and low-frequency MOE knowledge to adapt to
remote sensing images with significant content variations. Quantitative and
qualitative experiments over multiple datasets demonstrate that our method
performs the best against other state-of-the-art ones and comprises a strong
generalization ability for real-world scenes. Code will be made publicly at
\url{https://github.com/alexhe101/FAME-Net}.
