  Matching-based networks have achieved state-of-the-art performance for video
object segmentation (VOS) tasks by storing every-k frames in an external memory
bank for future inference. Storing the intermediate frames' predictions
provides the network with richer cues for segmenting an object in the current
frame. However, the size of the memory bank gradually increases with the length
of the video, which slows down inference speed and makes it impractical to
handle arbitrary length videos.
  This paper proposes an adaptive memory bank strategy for matching-based
networks for semi-supervised video object segmentation (VOS) that can handle
videos of arbitrary length by discarding obsolete features. Features are
indexed based on their importance in the segmentation of the objects in
previous frames. Based on the index, we discard unimportant features to
accommodate new features. We present our experiments on DAVIS 2016, DAVIS 2017,
and Youtube-VOS that demonstrate that our method outperforms state-of-the-art
that employ first-and-latest strategy with fixed-sized memory banks and
achieves comparable performance to the every-k strategy with increasing-sized
memory banks. Furthermore, experiments show that our method increases inference
speed by up to 80% over the every-k and 35% over first-and-latest strategies.
