  High-accuracy per-pixel depth is vital for computational photography, so
smartphones now have multimodal camera systems with time-of-flight (ToF) depth
sensors and multiple color cameras. However, producing accurate high-resolution
depth is still challenging due to the low resolution and limited active
illumination power of ToF sensors. Fusing RGB stereo and ToF information is a
promising direction to overcome these issues, but a key problem remains: to
provide high-quality 2D RGB images, the main color sensor's lens is optically
stabilized, resulting in an unknown pose for the floating lens that breaks the
geometric relationships between the multimodal image sensors. Leveraging ToF
depth estimates and a wide-angle RGB camera, we design an automatic calibration
technique based on dense 2D/3D matching that can estimate camera extrinsic,
intrinsic, and distortion parameters of a stabilized main RGB sensor from a
single snapshot. This lets us fuse stereo and ToF cues via a correlation
volume. For fusion, we apply deep learning via a real-world training dataset
with depth supervision estimated by a neural reconstruction method. For
evaluation, we acquire a test dataset using a commercial high-power depth
camera and show that our approach achieves higher accuracy than existing
baselines.
