  Visual object tracking plays a critical role in visual-based autonomous
systems, as it aims to estimate the position and size of the object of interest
within a live video. Despite significant progress made in this field,
state-of-the-art (SOTA) trackers often fail when faced with adversarial
perturbations in the incoming frames. This can lead to significant robustness
and security issues when these trackers are deployed in the real world. To
achieve high accuracy on both clean and adversarial data, we propose building a
spatial-temporal continuous representation using the semantic text guidance of
the object of interest. This novel continuous representation enables us to
reconstruct incoming frames to maintain semantic and appearance consistency
with the object of interest and its clean counterparts. As a result, our
proposed method successfully defends against different SOTA adversarial
tracking attacks while maintaining high accuracy on clean data. In particular,
our method significantly increases tracking accuracy under adversarial attacks
with around 90% relative improvement on UAV123, which is even higher than the
accuracy on clean data.
