  Nearly all models for object tracking with artificial neural networks depend
on appearance features extracted from a "backbone" architecture, designed for
object recognition. Indeed, significant progress on object tracking has been
spurred by introducing backbones that are better able to discriminate objects
by their appearance. However, extensive neurophysiology and psychophysics
evidence suggests that biological visual systems track objects using both
appearance and motion features. Here, we introduce $\textit{PathTracker}$, a
visual challenge inspired by cognitive psychology, which tests the ability of
observers to learn to track objects solely by their motion. We find that
standard 3D-convolutional deep network models struggle to solve this task when
clutter is introduced into the generated scenes, or when objects travel long
distances. This challenge reveals that tracing the path of object motion is a
blind spot of feedforward neural networks. We expect that strategies for
appearance-free object tracking from biological vision can inspire solutions
these failures of deep neural networks.
