  Recent text-guided generation of individual 3D object has achieved great
success using diffusion priors. However, these methods are not suitable for
object insertion and replacement tasks as they do not consider the background,
leading to illumination mismatches within the environment. To bridge the gap,
we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian
Splatting (3DGS) representation. Our key observation is that inpainting by the
state-of-the-art conditional 2D diffusion model is consistent with background
in lighting. To leverage the prior knowledge from the well-trained diffusion
models for 3D object generation, our approach employs a coarse-to-fine
objection optimization pipeline with inpainted views. In the first coarse step,
we achieve image-to-3D lifting given an ideal inpainted view. The process
employs 3D-aware diffusion prior from a view-conditioned diffusion model, which
preserves illumination present in the conditioning image. To acquire an ideal
inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a
single view that best represents the scene illumination in target region. In
the second Texture Enhancement step, we introduce a novel Depth-guided
Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and
texture details with the inpainting diffusion prior, beyond the scope of the
3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only
provides fine-grained texture enhancement, but also urges optimization to
respect scene lighting. Our approach efficiently achieves local editing with
global illumination consistency without explicitly modeling light transport. We
demonstrate robustness of our method by evaluating editing in real scenes
containing explicit highlight and shadows, and compare against the
state-of-the-art text-to-3D editing methods.
