  Tracking the 6D pose of objects in video sequences is important for robot
manipulation. Most prior efforts, however, often assume that the target
object's CAD model, at least at a category-level, is available for offline
training or during online template matching. This work proposes BundleTrack, a
general framework for 6D pose tracking of novel objects, which does not depend
upon 3D models, either at the instance or category-level. It leverages the
complementary attributes of recent advances in deep learning for segmentation
and robust feature extraction, as well as memory-augmented pose graph
optimization for spatiotemporal consistency. This enables long-term, low-drift
tracking under various challenging scenarios, including significant occlusions
and object motions. Comprehensive experiments given two public benchmarks
demonstrate that the proposed approach significantly outperforms state-of-art,
category-level 6D tracking or dynamic SLAM methods. When compared against
state-of-art methods that rely on an object instance CAD model, comparable
performance is achieved, despite the proposed method's reduced information
requirements. An efficient implementation in CUDA provides a real-time
performance of 10Hz for the entire framework. Code is available at:
https://github.com/wenbowen123/BundleTrack
