  Depth maps are used in a wide range of applications from 3D rendering to 2D
image effects such as Bokeh. However, those predicted by single image depth
estimation (SIDE) models often fail to capture isolated holes in objects and/or
have inaccurate boundary regions. Meanwhile, high-quality masks are much easier
to obtain, using commercial auto-masking tools or off-the-shelf methods of
segmentation and matting or even by manual editing. Hence, in this paper, we
formulate a novel problem of mask-guided depth refinement that utilizes a
generic mask to refine the depth prediction of SIDE models. Our framework
performs layered refinement and inpainting/outpainting, decomposing the depth
map into two separate layers signified by the mask and the inverse mask. As
datasets with both depth and mask annotations are scarce, we propose a
self-supervised learning scheme that uses arbitrary masks and RGB-D datasets.
We empirically show that our method is robust to different types of masks and
initial depth predictions, accurately refining depth values in inner and outer
mask boundary regions. We further analyze our model with an ablation study and
demonstrate results on real applications. More information can be found at
https://sooyekim.github.io/MaskDepth/ .
