  Learning based 6D object pose estimation methods rely on computing large
intermediate pose representations and/or iteratively refining an initial
estimation with a slow render-compare pipeline. This paper introduces a novel
method we call Cascaded Pose Refinement Transformers, or CRT-6D. We replace the
commonly used dense intermediate representation with a sparse set of features
sampled from the feature pyramid we call OSKFs(Object Surface Keypoint
Features) where each element corresponds to an object keypoint. We employ
lightweight deformable transformers and chain them together to iteratively
refine proposed poses over the sampled OSKFs. We achieve inference runtimes 2x
faster than the closest real-time state of the art methods while supporting up
to 21 objects on a single model. We demonstrate the effectiveness of CRT-6D by
performing extensive experiments on the LM-O and YCBV datasets. Compared to
real-time methods, we achieve state of the art on LM-O and YCB-V, falling
slightly behind methods with inference runtimes one order of magnitude higher.
The source code is available at: https://github.com/PedroCastro/CRT-6D
