  This paper proposes a cross-modal distillation framework, PartDistill, which
transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D
shape part segmentation. PartDistill addresses three major challenges in this
task: the lack of 3D segmentation in invisible or undetected regions in the 2D
projections, inconsistent 2D predictions by VLMs, and the lack of knowledge
accumulation across different 3D shapes. PartDistill consists of a teacher
network that uses a VLM to make 2D predictions and a student network that
learns from the 2D predictions while extracting geometrical features from
multiple 3D shapes to carry out 3D part segmentation. A bi-directional
distillation, including forward and backward distillations, is carried out
within the framework, where the former forward distills the 2D predictions to
the student network, and the latter improves the quality of the 2D predictions,
which subsequently enhances the final 3D segmentation. Moreover, PartDistill
can exploit generative models that facilitate effortless 3D shape creation for
generating knowledge sources to be distilled. Through extensive experiments,
PartDistill boosts the existing methods with substantial margins on widely used
ShapeNetPart and PartNetE datasets, by more than 15% and 12% higher mIoU
scores, respectively. The code for this work is available at
https://github.com/ardianumam/PartDistill.
