  Nowadays, the need for user editing in a 3D scene has rapidly increased due
to the development of AR and VR technology. However, the existing 3D scene
completion task (and datasets) cannot suit the need because the missing regions
in scenes are generated by the sensor limitation or object occlusion. Thus, we
present a novel task named free-form 3D scene inpainting. Unlike scenes in
previous 3D completion datasets preserving most of the main structures and
hints of detailed shapes around missing regions, the proposed inpainting
dataset, FF-Matterport, contains large and diverse missing regions formed by
our free-form 3D mask generation algorithm that can mimic human drawing
trajectories in 3D space. Moreover, prior 3D completion methods cannot perform
well on this challenging yet practical task, simply interpolating nearby
geometry and color context. Thus, a tailored dual-stream GAN method is
proposed. First, our dual-stream generator, fusing both geometry and color
information, produces distinct semantic boundaries and solves the interpolation
issue. To further enhance the details, our lightweight dual-stream
discriminator regularizes the geometry and color edges of the predicted scenes
to be realistic and sharp. We conducted experiments with the proposed
FF-Matterport dataset. Qualitative and quantitative results validate the
superiority of our approach over existing scene completion methods and the
efficacy of all proposed components.
