  Due to inevitable noises introduced during scanning and quantization, 3D
reconstruction via RGB-D sensors suffers from errors both in geometry and
texture, leading to artifacts such as camera drifting, mesh distortion, texture
ghosting, and blurriness. Given an imperfect reconstructed 3D model, most
previous methods have focused on the refinement of either geometry, texture, or
camera pose. Or different optimization schemes and objectives for optimizing
each component have been used in previous joint optimization methods, forming a
complicated system. In this paper, we propose a novel optimization approach
based on differentiable rendering, which integrates the optimization of camera
pose, geometry, and texture into a unified framework by enforcing consistency
between the rendered results and the corresponding RGB-D inputs. Based on the
unified framework, we introduce a joint optimization approach to fully exploit
the inter-relationships between geometry, texture, and camera pose, and
describe an adaptive interleaving strategy to improve optimization stability
and efficiency. Using differentiable rendering, an image-level adversarial loss
is applied to further improve the 3D model, making it more photorealistic.
Experiments on synthetic and real data using quantitative and qualitative
evaluation demonstrated the superiority of our approach in recovering both
fine-scale geometry and high-fidelity texture.Code is available at
https://adjointopti.github.io/adjoin.github.io/.
