  Virtual and augmented reality technologies have seen significant growth in
the past few years. A key component of such systems is the ability to track the
pose of head mounted displays and controllers in 3D space. We tackle the
problem of efficient 6-DoF tracking of a handheld controller from egocentric
camera perspectives. We collected the HMD Controller dataset which consist of
over 540,000 stereo image pairs labelled with the full 6-DoF pose of the
handheld controller. Our proposed SSD-AF-Stereo3D model achieves a mean average
error of 33.5 millimeters in 3D keypoint prediction and is used in conjunction
with an IMU sensor on the controller to enable 6-DoF tracking. We also present
results on approaches for model based full 6-DoF tracking. All our models
operate under the strict constraints of real time mobile CPU inference.
