  Zero-shot 6D object pose estimation involves the detection of novel objects
with their 6D poses in cluttered scenes, presenting significant challenges for
model generalizability. Fortunately, the recent Segment Anything Model (SAM)
has showcased remarkable zero-shot transfer performance, which provides a
promising solution to tackle this task. Motivated by this, we introduce SAM-6D,
a novel framework designed to realize the task through two steps, including
instance segmentation and pose estimation. Given the target objects, SAM-6D
employs two dedicated sub-networks, namely Instance Segmentation Model (ISM)
and Pose Estimation Model (PEM), to perform these steps on cluttered RGB-D
images. ISM takes SAM as an advanced starting point to generate all possible
object proposals and selectively preserves valid ones through meticulously
crafted object matching scores in terms of semantics, appearance and geometry.
By treating pose estimation as a partial-to-partial point matching problem, PEM
performs a two-stage point matching process featuring a novel design of
background tokens to construct dense 3D-3D correspondence, ultimately yielding
the pose estimates. Without bells and whistles, SAM-6D outperforms the existing
methods on the seven core datasets of the BOP Benchmark for both instance
segmentation and pose estimation of novel objects.
