  Image view synthesis has seen great success in reconstructing photorealistic
visuals, thanks to deep learning and various novel representations. The next
key step in immersive virtual experiences is view synthesis of dynamic scenes.
However, several challenges exist due to the lack of high-quality training
datasets, and the additional time dimension for videos of dynamic scenes. To
address this issue, we introduce a multi-view video dataset, captured with a
custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes
showing various visual effects and human interactions in outdoor scenes. We
develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable
view extrapolation from binocular videos of dynamic scenes, captured by static
cameras. Our algorithm addresses the temporal inconsistency of disocclusions by
identifying the error-prone areas with a 3D mask volume, and replaces them with
static background observed throughout the video. Our method enables
manipulation in 3D space as opposed to simple 2D masks, We demonstrate better
temporal stability than frame-by-frame static view synthesis methods, or those
that use 2D masks. The resulting view synthesis videos show minimal flickering
artifacts and allow for larger translational movements.
