  Estimating 6D object pose from an RGB image is important for many real-world
applications such as autonomous driving and robotic grasping. Recent deep
learning models have achieved significant progress on this task but their
robustness received little research attention. In this work, for the first
time, we study adversarial samples that can fool deep learning models with
imperceptible perturbations to input image. In particular, we propose a Unified
6D pose estimation Attack, namely U6DA, which can successfully attack several
state-of-the-art (SOTA) deep learning models for 6D pose estimation. The key
idea of our U6DA is to fool the models to predict wrong results for object
instance localization and shape that are essential for correct 6D pose
estimation. Specifically, we explore a transfer-based black-box attack to 6D
pose estimation. We design the U6DA loss to guide the generation of adversarial
examples, the loss aims to shift the segmentation attention map away from its
original position. We show that the generated adversarial samples are not only
effective for direct 6D pose estimation models, but also are able to attack
two-stage models regardless of their robust RANSAC modules. Extensive
experiments were conducted to demonstrate the effectiveness, transferability,
and anti-defense capability of our U6DA on large-scale public benchmarks. We
also introduce a new U6DA-Linemod dataset for robustness study of the 6D pose
estimation task. Our codes and dataset will be available at
\url{https://github.com/cuge1995/U6DA}.
