  One-shot Video Object Segmentation~(VOS) is the task of pixel-wise tracking
an object of interest within a video sequence, where the segmentation mask of
the first frame is given at inference time. In recent years, Recurrent Neural
Networks~(RNNs) have been widely used for VOS tasks, but they often suffer from
limitations such as drift and error propagation. In this work, we study an
RNN-based architecture and address some of these issues by proposing a hybrid
sequence-to-sequence architecture named HS2S, utilizing a dual mask propagation
strategy that allows incorporating the information obtained from correspondence
matching. Our experiments show that augmenting the RNN with correspondence
matching is a highly effective solution to reduce the drift problem. The
additional information helps the model to predict more accurate masks and makes
it robust against error propagation. We evaluate our HS2S model on the
DAVIS2017 dataset as well as Youtube-VOS. On the latter, we achieve an
improvement of 11.2pp in the overall segmentation accuracy over RNN-based
state-of-the-art methods in VOS. We analyze our model's behavior in challenging
cases such as occlusion and long sequences and show that our hybrid
architecture significantly enhances the segmentation quality in these difficult
scenarios.
