  Commercial depth sensors usually generate noisy and missing depths,
especially on specular and transparent objects, which poses critical issues to
downstream depth or point cloud-based tasks. To mitigate this problem, we
propose a powerful RGBD fusion network, SwinDRNet, for depth restoration. We
further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach
to simulate an active stereo depth system using physically based rendering and
generate a large-scale synthetic dataset that contains 130K photorealistic RGB
images along with their simulated depths carrying realistic sensor noises. To
evaluate depth restoration methods, we also curate a real-world dataset, namely
STD, that captures 30 cluttered scenes composed of 50 objects with different
materials from specular, transparent, to diffuse. Experiments demonstrate that
the proposed DREDS dataset bridges the sim-to-real domain gap such that,
trained on DREDS, our SwinDRNet can seamlessly generalize to other real depth
datasets, e.g. ClearGrasp, and outperform the competing methods on depth
restoration with a real-time speed. We further show that our depth restoration
effectively boosts the performance of downstream tasks, including
category-level pose estimation and grasping tasks. Our data and code are
available at https://github.com/PKU-EPIC/DREDS
