  Novel view synthesis of remote sensing scenes is of great significance for
scene visualization, human-computer interaction, and various downstream
applications. Despite the recent advances in computer graphics and
photogrammetry technology, generating novel views is still challenging
particularly for remote sensing images due to its high complexity, view
sparsity and limited view-perspective variations. In this paper, we propose a
novel remote sensing view synthesis method by leveraging the recent advances in
implicit neural representations. Considering the overhead and far depth imaging
of remote sensing images, we represent the 3D space by combining implicit
multiplane images (MPI) representation and deep neural networks. The 3D scene
is reconstructed under a self-supervised optimization paradigm through a
differentiable multiplane renderer with multi-view input constraints. Images
from any novel views thus can be freely rendered on the basis of the
reconstructed model. As a by-product, the depth maps corresponding to the given
viewpoint can be generated along with the rendering output. We refer to our
method as Implicit Multiplane Images (ImMPI). To further improve the view
synthesis under sparse-view inputs, we explore the learning-based
initialization of remote sensing 3D scenes and proposed a neural network based
Prior extractor to accelerate the optimization process. In addition, we propose
a new dataset for remote sensing novel view synthesis with multi-view
real-world google earth images. Extensive experiments demonstrate the
superiority of the ImMPI over previous state-of-the-art methods in terms of
reconstruction accuracy, visual fidelity, and time efficiency. Ablation
experiments also suggest the effectiveness of our methodology design. Our
dataset and code can be found at https://github.com/wyc-Chang/ImMPI
