  This paper proposes a Robust and Efficient Memory Network, referred to as
REMN, for studying semi-supervised video object segmentation (VOS).
Memory-based methods have recently achieved outstanding VOS performance by
performing non-local pixel-wise matching between the query and memory. However,
these methods have two limitations. 1) Non-local matching could cause
distractor objects in the background to be incorrectly segmented. 2) Memory
features with high temporal redundancy consume significant computing resources.
For limitation 1, we introduce a local attention mechanism that tackles the
background distraction by enhancing the features of foreground objects with the
previous mask. For limitation 2, we first adaptively decide whether to update
the memory features depending on the variation of foreground objects to reduce
temporal redundancy. Second, we employ a dynamic memory bank, which uses a
lightweight and differentiable soft modulation gate to decide how many memory
features need to be removed in the temporal dimension. Experiments demonstrate
that our REMN achieves state-of-the-art results on DAVIS 2017, with a
$\mathcal{J\&F}$ score of 86.3% and on YouTube-VOS 2018, with a $\mathcal{G}$
over mean of 85.5%. Furthermore, our network shows a high inference speed of
25+ FPS and uses relatively few computing resources.
