  Indoor scene modification has emerged as a prominent area within computer
vision, particularly for its applications in Augmented Reality (AR) and Virtual
Reality (VR). Traditional methods often rely on pre-existing object databases
and predetermined object positions, limiting their flexibility and adaptability
to new scenarios. In response to this challenge, we present a novel end-to-end
multi-modal deep neural network capable of generating point cloud objects
seamlessly integrated with their surroundings, driven by textual instructions.
Our model revolutionizes scene modification by enabling the creation of new
environments with previously unseen object layouts, eliminating the need for
pre-stored CAD models. Leveraging Point-E as our generative model, we introduce
innovative techniques such as quantized position prediction and Top-K
estimation to address the issue of false negatives resulting from ambiguous
language descriptions. Furthermore, we conduct comprehensive evaluations to
showcase the diversity of generated objects, the efficacy of textual
instructions, and the quantitative metrics, affirming the realism and
versatility of our model in generating indoor objects. To provide a holistic
assessment, we incorporate visual grounding as an additional metric, ensuring
the quality and coherence of the scenes produced by our model. Through these
advancements, our approach not only advances the state-of-the-art in indoor
scene modification but also lays the foundation for future innovations in
immersive computing and digital environment creation.
