  This paper proposes ConsistDreamer - a novel framework that lifts 2D
diffusion models with 3D awareness and 3D consistency, thus enabling
high-fidelity instruction-guided scene editing. To overcome the fundamental
limitation of missing 3D consistency in 2D diffusion models, our key insight is
to introduce three synergetic strategies that augment the input of the 2D
diffusion model to become 3D-aware and to explicitly enforce 3D consistency
during the training process. Specifically, we design surrounding views as
context-rich input for the 2D diffusion model, and generate 3D-consistent,
structured noise instead of image-independent noise. Moreover, we introduce
self-supervised consistency-enforcing training within the per-scene editing
procedure. Extensive evaluation shows that our ConsistDreamer achieves
state-of-the-art performance for instruction-guided scene editing across
various scenes and editing instructions, particularly in complicated
large-scale indoor scenes from ScanNet++, with significantly improved sharpness
and fine-grained textures. Notably, ConsistDreamer stands as the first work
capable of successfully editing complex (e.g., plaid/checkered) patterns. Our
project page is at immortalco.github.io/ConsistDreamer.
