  We introduce a novel network, called CO-attention Siamese Network (COSNet),
to address the unsupervised video object segmentation task from a holistic
view. We emphasize the importance of inherent correlation among video frames
and incorporate a global co-attention mechanism to improve further the
state-of-the-art deep learning based solutions that primarily focus on learning
discriminative foreground representations over appearance and motion in
short-term temporal segments. The co-attention layers in our network provide
efficient and competent stages for capturing global correlations and scene
context by jointly computing and appending co-attention responses into a joint
feature space. We train COSNet with pairs of video frames, which naturally
augments training data and allows increased learning capacity. During the
segmentation stage, the co-attention model encodes useful information by
processing multiple reference frames together, which is leveraged to infer the
frequently reappearing and salient foreground objects better. We propose a
unified and end-to-end trainable framework where different co-attention
variants can be derived for mining the rich context within videos. Our
extensive experiments over three large benchmarks manifest that COSNet
outperforms the current alternatives by a large margin.
