  Image-based 3D Virtual Try-ON (VTON) aims to sculpt the 3D human according to
person and clothes images, which is data-efficient (i.e., getting rid of
expensive 3D data) but challenging. Recent text-to-3D methods achieve
remarkable improvement in high-fidelity 3D human generation, demonstrating its
potential for 3D virtual try-on. Inspired by the impressive success of
personalized diffusion models (e.g., Dreambooth and LoRA) for 2D VTON, it is
straightforward to achieve 3D VTON by integrating the personalization technique
into the diffusion-based text-to-3D framework. However, employing the
personalized module in a pre-trained diffusion model (e.g., StableDiffusion
(SD)) would degrade the model's capability for multi-view or multi-domain
synthesis, which is detrimental to the geometry and texture optimization guided
by Score Distillation Sampling (SDS) loss. In this work, we propose a novel
customizing 3D human try-on model, named \textbf{DreamVTON}, to separately
optimize the geometry and texture of the 3D human. Specifically, a personalized
SD with multi-concept LoRA is proposed to provide the generative prior about
the specific person and clothes, while a Densepose-guided ControlNet is
exploited to guarantee consistent prior about body pose across various camera
views. Besides, to avoid the inconsistent multi-view priors from the
personalized SD dominating the optimization, DreamVTON introduces a
template-based optimization mechanism, which employs mask templates for
geometry shape learning and normal/RGB templates for geometry/texture details
learning. Furthermore, for the geometry optimization phase, DreamVTON
integrates a normal-style LoRA into personalized SD to enhance normal map
generative prior, facilitating smooth geometry modeling.
