  Generating 3D vehicle assets from in-the-wild observations is crucial to
autonomous driving. Existing image-to-3D methods cannot well address this
problem because they learn generation merely from image RGB information without
a deeper understanding of in-the-wild vehicles (such as car models,
manufacturers, etc.). This leads to their poor zero-shot prediction capability
to handle real-world observations with occlusion or tricky viewing angles. To
solve this problem, in this work, we propose VQA-Diff, a novel framework that
leverages in-the-wild vehicle images to create photorealistic 3D vehicle assets
for autonomous driving. VQA-Diff exploits the real-world knowledge inherited
from the Large Language Model in the Visual Question Answering (VQA) model for
robust zero-shot prediction and the rich image prior knowledge in the Diffusion
model for structure and appearance generation. In particular, we utilize a
multi-expert Diffusion Models strategy to generate the structure information
and employ a subject-driven structure-controlled generation mechanism to model
appearance information. As a result, without the necessity to learn from a
large-scale image-to-3D vehicle dataset collected from the real world, VQA-Diff
still has a robust zero-shot image-to-novel-view generation ability. We conduct
experiments on various datasets, including Pascal 3D+, Waymo, and Objaverse, to
demonstrate that VQA-Diff outperforms existing state-of-the-art methods both
qualitatively and quantitatively.
