  Deep convolutional neural network (DCNN) has been successfully applied to
depth map super-resolution and outperforms existing methods by a wide margin.
However, there still exist two major issues with these DCNN based depth map
super-resolution methods that hinder the performance: i) The low-resolution
depth maps either need to be up-sampled before feeding into the network or
substantial deconvolution has to be used; and ii) The supervision
(high-resolution depth maps) is only applied at the end of the network, thus it
is difficult to handle large up-sampling factors, such as $\times 8, \times
16$. In this paper, we propose a new framework to tackle the above problems.
First, we propose to represent the task of depth map super-resolution as a
series of novel view synthesis sub-tasks. The novel view synthesis sub-task
aims at generating (synthesizing) a depth map from different camera pose, which
could be learned in parallel. Second, to handle large up-sampling factors, we
present a deeply supervised network structure to enforce strong supervision in
each stage of the network. Third, a multi-scale fusion strategy is proposed to
effectively exploit the feature maps at different scales and handle the
blocking effect. In this way, our proposed framework could deal with
challenging depth map super-resolution efficiently under large up-sampling
factors (e.g. $\times 8, \times 16$). Our method only uses the low-resolution
depth map as input, and the support of color image is not needed, which greatly
reduces the restriction of our method. Extensive experiments on various
benchmarking datasets demonstrate the superiority of our method over current
state-of-the-art depth map super-resolution methods.
