  Feature tracking is the building block of many applications such as visual
odometry, augmented reality, and target tracking. Unfortunately, the
state-of-the-art vision-based tracking algorithms fail in surgical images due
to the challenges imposed by the nature of such environments. In this paper, we
proposed a novel and unified deep learning-based approach that can learn how to
track features reliably as well as learn how to detect such reliable features
for tracking purposes. The proposed network dubbed as Deep-PT, consists of a
tracker network which is a convolutional neural network simulating
cross-correlation in terms of deep learning and two fully connected networks
that operate on the output of intermediate layers of the tracker to detect
features and predict trackability of the detected points. The ability to detect
features based on the capabilities of the tracker distinguishes the proposed
method from previous algorithms used in this area and improves the robustness
of the algorithms against dynamics of the scene. The network is trained using
multiple datasets due to the lack of specialized dataset for feature tracking
datasets and extensive comparisons are conducted to compare the accuracy of
Deep-PT against recent pixel tracking algorithms. As the experiments suggest,
the proposed deep architecture deliberately learns what to track and how to
track and outperforms the state-of-the-art methods.
