  We tackle a new problem of semantic view synthesis -- generating
free-viewpoint rendering of a synthesized scene using a semantic label map as
input. We build upon recent advances in semantic image synthesis and view
synthesis for handling photographic image content generation and view
extrapolation. Direct application of existing image/view synthesis methods,
however, results in severe ghosting/blurry artifacts. To address the drawbacks,
we propose a two-step approach. First, we focus on synthesizing the color and
depth of the visible surface of the 3D scene. We then use the synthesized color
and depth to impose explicit constraints on the multiple-plane image (MPI)
representation prediction process. Our method produces sharp contents at the
original view and geometrically consistent renderings across novel viewpoints.
The experiments on numerous indoor and outdoor images show favorable results
against several strong baselines and validate the effectiveness of our
approach.
