  We propose a learning-based approach for novel view synthesis for
multi-camera 360$^{\circ}$ panorama capture rigs. Previous work constructs RGBD
panoramas from such data, allowing for view synthesis with small amounts of
translation, but cannot handle the disocclusions and view-dependent effects
that are caused by large translations. To address this issue, we present a
novel scene representation - Multi Depth Panorama (MDP) - that consists of
multiple RGBD$\alpha$ panoramas that represent both scene geometry and
appearance. We demonstrate a deep neural network-based method to reconstruct
MDPs from multi-camera 360$^{\circ}$ images. MDPs are more compact than
previous 3D scene representations and enable high-quality, efficient new view
rendering. We demonstrate this via experiments on both synthetic and real data
and comparisons with previous state-of-the-art methods spanning both
learning-based approaches and classical RGBD-based methods.
