  Depth (D) indicates occlusion and is less sensitive to illumination changes,
which make depth attractive modality for Visual Object Tracking (VOT). Depth is
used in RGBD object tracking where the best trackers are deep RGB trackers with
additional heuristic using depth maps. There are two potential reasons for the
heuristics: 1) the lack of large RGBD tracking datasets to train deep RGBD
trackers and 2) the long-term evaluation protocol of VOT RGBD that benefits
from heuristics such as depth-based occlusion detection. In this work, we study
how far D-only tracking can go if trained with large amounts of depth data. To
compensate the lack of depth data, we generate depth maps for tracking. We
train a "Depth-DiMP" from the scratch with the generated data and fine-tune it
with the available small RGBD tracking datasets. The depth-only DiMP achieves
good accuracy in depth-only tracking and combined with the original RGB DiMP
the end-to-end trained RGBD-DiMP outperforms the recent VOT 2020 RGBD winners.
