  Multiple object tracking and segmentation requires detecting, tracking, and
segmenting objects belonging to a set of given classes. Most approaches only
exploit the temporal dimension to address the association problem, while
relying on single frame predictions for the segmentation mask itself. We
propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich
spatio-temporal information for online multiple object tracking and
segmentation. PCAN first distills a space-time memory into a set of prototypes
and then employs cross-attention to retrieve rich information from the past
frames. To segment each object, PCAN adopts a prototypical appearance module to
learn a set of contrastive foreground and background prototypes, which are then
propagated over time. Extensive experiments demonstrate that PCAN outperforms
current video instance tracking and segmentation competition winners on both
Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and
two-stage segmentation frameworks. Code and video resources are available at
http://vis.xyz/pub/pcan.
