  Automatic furniture layout is long desired for convenient interior design.
Leveraging the remarkable visual reasoning capabilities of multimodal large
language models (MLLMs), recent methods address layout generation in a static
manner, lacking the feedback-driven refinement essential for interactive user
engagement. We introduce Chat2Layout, a novel interactive furniture layout
generation system that extends the functionality of MLLMs into the realm of
interactive layout design. To achieve this, we establish a unified
vision-question paradigm for in-context learning, enabling seamless
communication with MLLMs to steer their behavior without altering model
weights. Within this framework, we present a novel training-free visual
prompting mechanism. This involves a visual-text prompting technique that
assist MLLMs in reasoning about plausible layout plans, followed by an
Offline-to-Online search (O2O-Search) method, which automatically identifies
the minimal set of informative references to provide exemplars for visual-text
prompting. By employing an agent system with MLLMs as the core controller, we
enable bidirectional interaction. The agent not only comprehends the 3D
environment and user requirements through linguistic and visual perception but
also plans tasks and reasons about actions to generate and arrange furniture
within the virtual space. Furthermore, the agent iteratively updates based on
visual feedback from execution results. Experimental results demonstrate that
our approach facilitates language-interactive generation and arrangement for
diverse and complex 3D furniture.
