  Existing monocular depth estimation methods have achieved excellent
robustness in diverse scenes, but they can only retrieve affine-invariant
depth, up to an unknown scale and shift. However, in some video-based scenarios
such as video depth estimation and 3D scene reconstruction from a video, the
unknown scale and shift residing in per-frame prediction may cause the depth
inconsistency. To solve this problem, we propose a locally weighted linear
regression method to recover the scale and shift with very sparse anchor
points, which ensures the scale consistency along consecutive frames. Extensive
experiments show that our method can boost the performance of existing
state-of-the-art approaches by 50% at most over several zero-shot benchmarks.
Besides, we merge over 6.3 million RGBD images to train strong and robust depth
models. Our produced ResNet50-backbone model even outperforms the
state-of-the-art DPT ViT-Large model. Combining with geometry-based
reconstruction methods, we formulate a new dense 3D scene reconstruction
pipeline, which benefits from both the scale consistency of sparse points and
the robustness of monocular methods. By performing the simple per-frame
prediction over a video, the accurate 3D scene shape can be recovered.
