  In this paper we present a novel approach for depth map enhancement from an
RGB-D video sequence. The basic idea is to exploit the shading information in
the color image. Instead of making assumption about surface albedo or
controlled object motion and lighting, we use the lighting variations
introduced by casual object movement. We are effectively calculating
photometric stereo from a moving object under natural illuminations. The key
technical challenge is to establish correspondences over the entire image set.
We therefore develop a lighting insensitive robust pixel matching technique
that out-performs optical flow method in presence of lighting variations. In
addition we present an expectation-maximization framework to recover the
surface normal and albedo simultaneously, without any regularization term. We
have validated our method on both synthetic and real datasets to show its
superior performance on both surface details recovery and intrinsic
decomposition.
