  Pansharpening refers to the fusion of a panchromatic image with a high
spatial resolution and a multispectral image with a low spatial resolution,
aiming to obtain a high spatial resolution multispectral image. In this paper,
we propose a novel deep neural network architecture with level-domain based
loss function for pansharpening by taking into account the following
double-type structures, \emph{i.e.,} double-level, double-branch, and
double-direction, called as triple-double network (TDNet). By using the
structure of TDNet, the spatial details of the panchromatic image can be fully
exploited and utilized to progressively inject into the low spatial resolution
multispectral image, thus yielding the high spatial resolution output. The
specific network design is motivated by the physical formula of the traditional
multi-resolution analysis (MRA) methods. Hence, an effective MRA fusion module
is also integrated into the TDNet. Besides, we adopt a few ResNet blocks and
some multi-scale convolution kernels to deepen and widen the network to
effectively enhance the feature extraction and the robustness of the proposed
TDNet. Extensive experiments on reduced- and full-resolution datasets acquired
by WorldView-3, QuickBird, and GaoFen-2 sensors demonstrate the superiority of
the proposed TDNet compared with some recent state-of-the-art pansharpening
approaches. An ablation study has also corroborated the effectiveness of the
proposed approach.
