  Time-of-Flight (ToF) depth sensing camera is able to obtain depth maps at a
high frame rate. However, its low resolution and sensitivity to the noise are
always a concern. A popular solution is upsampling the obtained noisy low
resolution depth map with the guidance of the companion high resolution color
image. However, due to the constrains in the existing upsampling models, the
high resolution depth map obtained in such way may suffer from either texture
copy artifacts or blur of depth discontinuity. In this paper, a novel
optimization framework is proposed with the brand new data term and smoothness
term. The comprehensive experiments using both synthetic data and real data
show that the proposed method well tackles the problem of texture copy
artifacts and blur of depth discontinuity. It also demonstrates sufficient
robustness to the noise. Moreover, a data driven scheme is proposed to
adaptively estimate the parameter in the upsampling optimization framework. The
encouraging performance is maintained even in the case of large upsampling e.g.
$8\times$ and $16\times$.
