  In this paper, we study the representation of the shape and pose of objects
using their keypoints. Therefore, we propose an end-to-end method that
simultaneously detects 2D keypoints from an image and lifts them to 3D. The
proposed method learns both 2D detection and 3D lifting only from 2D keypoints
annotations. In addition to being end-to-end from images to 3D keypoints, our
method also handles objects from multiple categories using a single neural
network. We use a Transformer-based architecture to detect the keypoints, as
well as to summarize the visual context of the image. This visual context
information is then used while lifting the keypoints to 3D, to allow
context-based reasoning for better performance. Our method can handle
occlusions as well as a wide variety of object classes. Our experiments on
three benchmarks demonstrate that our method performs better than the
state-of-the-art. Our source code will be made publicly available.
