  CAD model retrieval to real-world scene observations has shown strong promise
as a basis for 3D perception of objects and a clean, lightweight mesh-based
scene representation; however, current approaches to retrieve CAD models to a
query scan rely on expensive manual annotations of 1:1 associations of CAD-scan
objects, which typically contain strong lower-level geometric differences. We
thus propose a new weakly-supervised approach to retrieve semantically and
structurally similar CAD models to a query 3D scanned scene without requiring
any CAD-scan associations, and only object detection information as oriented
bounding boxes. Our approach leverages a fully-differentiable top-$k$ retrieval
layer, enabling end-to-end training guided by geometric and perceptual
similarity of the top retrieved CAD models to the scan queries. We demonstrate
that our weakly-supervised approach can outperform fully-supervised retrieval
methods on challenging real-world ScanNet scans, and maintain robustness for
unseen class categories, achieving significantly improved performance over
fully-supervised state of the art in zero-shot CAD retrieval.
