  Imagine trying to track one particular fruitfly in a swarm of hundreds.
Higher biological visual systems have evolved to track moving objects by
relying on both appearance and motion features. We investigate if
state-of-the-art deep neural networks for visual tracking are capable of the
same. For this, we introduce PathTracker, a synthetic visual challenge that
asks human observers and machines to track a target object in the midst of
identical-looking "distractor" objects. While humans effortlessly learn
PathTracker and generalize to systematic variations in task design,
state-of-the-art deep networks struggle. To address this limitation, we
identify and model circuit mechanisms in biological brains that are implicated
in tracking objects based on motion cues. When instantiated as a recurrent
network, our circuit model learns to solve PathTracker with a robust visual
strategy that rivals human performance and explains a significant proportion of
their decision-making on the challenge. We also show that the success of this
circuit model extends to object tracking in natural videos. Adding it to a
transformer-based architecture for object tracking builds tolerance to visual
nuisances that affect object appearance, resulting in a new state-of-the-art
performance on the large-scale TrackingNet object tracking challenge. Our work
highlights the importance of building artificial vision models that can help us
better understand human vision and improve computer vision.
