  Recently, image-to-3D approaches have achieved significant results with a
natural image as input. However, it is not always possible to access these
enriched color input samples in practical applications, where only sketches are
available. Existing sketch-to-3D researches suffer from limitations in broad
applications due to the challenges of lacking color information and multi-view
content. To overcome them, this paper proposes a novel generation paradigm
Sketch3D to generate realistic 3D assets with shape aligned with the input
sketch and color matching the textual description. Concretely, Sketch3D first
instantiates the given sketch in the reference image through the
shape-preserving generation process. Second, the reference image is leveraged
to deduce a coarse 3D Gaussian prior, and multi-view style-consistent guidance
images are generated based on the renderings of the 3D Gaussians. Finally,
three strategies are designed to optimize 3D Gaussians, i.e., structural
optimization via a distribution transfer mechanism, color optimization with a
straightforward MSE loss and sketch similarity optimization with a CLIP-based
geometric similarity loss. Extensive visual comparisons and quantitative
analysis illustrate the advantage of our Sketch3D in generating realistic 3D
assets while preserving consistency with the input.
