  Digital twin is a problem of augmenting real objects with their digital
counterparts. It can underpin a wide range of applications in augmented reality
(AR), autonomy, and UI/UX. A critical component in a good digital-twin system
is real-time, accurate 3D object tracking. Most existing works solve 3D object
tracking through the lens of robotic grasping, employ older generations of
depth sensors, and measure performance metrics that may not apply to other
digital-twin applications such as in AR. In this work, we create a novel RGB-D
dataset, called Digital Twin Tracking Dataset (DTTD), to enable further
research of the problem and extend potential solutions towards longer ranges
and mm localization accuracy. To reduce point cloud noise from the input
source, we select the latest Microsoft Azure Kinect as the state-of-the-art
time-of-flight (ToF) camera. In total, 103 scenes of 10 common off-the-shelf
objects with rich textures are recorded, with each frame annotated with a
per-pixel semantic segmentation and ground-truth object poses provided by a
commercial motion capturing system. Through extensive experiments with
model-level and dataset-level analysis, we demonstrate that DTTD can help
researchers develop future object tracking methods and analyze new challenges.
The dataset, data generation, annotation, and model evaluation pipeline are
made publicly available as open source code at:
https://github.com/augcog/DTTDv1.
