  We present ``just-in-time reconstruction" as real-time image-guided
inpainting of a map with arbitrary scale and sparsity to generate a fully dense
depth map for the image. In particular, our goal is to inpaint a sparse map ---
obtained from either a monocular visual SLAM system or a sparse sensor ---
using a single-view depth prediction network as a virtual depth sensor. We
adopt a fairly standard approach to data fusion, to produce a fused depth map
by performing inference over a novel fully-connected Conditional Random Field
(CRF) which is parameterized by the input depth maps and their pixel-wise
confidence weights. Crucially, we obtain the confidence weights that
parameterize the CRF model in a data-dependent manner via Convolutional Neural
Networks (CNNs) which are trained to model the conditional depth error
distributions given each source of input depth map and the associated RGB
image. Our CRF model penalises absolute depth error in its nodes and pairwise
scale-invariant depth error in its edges, and the confidence-based fusion
minimizes the impact of outlier input depth values on the fused result. We
demonstrate the flexibility of our method by real-time inpainting of ORB-SLAM,
Kinect, and LIDAR depth maps acquired both indoors and outdoors at arbitrary
scale and varied amount of irregular sparsity.
