  We present Modular interactive VOS (MiVOS) framework which decouples
interaction-to-mask and mask propagation, allowing for higher generalizability
and better performance. Trained separately, the interaction module converts
user interactions to an object mask, which is then temporally propagated by our
propagation module using a novel top-$k$ filtering strategy in reading the
space-time memory. To effectively take the user's intent into account, a novel
difference-aware module is proposed to learn how to properly fuse the masks
before and after each interaction, which are aligned with the target frames by
employing the space-time memory. We evaluate our method both qualitatively and
quantitatively with different forms of user interactions (e.g., scribbles,
clicks) on DAVIS to show that our method outperforms current state-of-the-art
algorithms while requiring fewer frame interactions, with the additional
advantage in generalizing to different types of user interactions. We
contribute a large-scale synthetic VOS dataset with pixel-accurate segmentation
of 4.8M frames to accompany our source codes to facilitate future research.
