  Text-based video segmentation aims to segment an actor in video sequences by
specifying the actor and its performing action with a textual query. Previous
methods fail to explicitly align the video content with the textual query in a
fine-grained manner according to the actor and its action, due to the problem
of \emph{semantic asymmetry}. The \emph{semantic asymmetry} implies that two
modalities contain different amounts of semantic information during the
multi-modal fusion process. To alleviate this problem, we propose a novel actor
and action modular network that individually localizes the actor and its action
in two separate modules. Specifically, we first learn the actor-/action-related
content from the video and textual query, and then match them in a symmetrical
manner to localize the target tube. The target tube contains the desired actor
and action which is then fed into a fully convolutional network to predict
segmentation masks of the actor. Our method also establishes the association of
objects cross multiple frames with the proposed temporal proposal aggregation
mechanism. This enables our method to segment the video effectively and keep
the temporal consistency of predictions. The whole model is allowed for joint
learning of the actor-action matching and segmentation, as well as achieves the
state-of-the-art performance for both single-frame segmentation and full video
segmentation on A2D Sentences and J-HMDB Sentences datasets.
