  Dominant pre-training work for video-text retrieval mainly adopt the
"dual-encoder" architectures to enable efficient retrieval, where two separate
encoders are used to contrast global video and text representations, but ignore
detailed local semantics. The recent success of image BERT pre-training with
masked visual modeling that promotes the learning of local visual context,
motivates a possible solution to address the above limitation. In this work, we
for the first time investigate masked visual modeling in video-text
pre-training with the "dual-encoder" architecture. We perform Masked visual
modeling with Injected LanguagE Semantics (MILES) by employing an extra
snapshot video encoder as an evolving "tokenizer" to produce reconstruction
targets for masked video patch prediction. Given the corrupted video, the video
encoder is trained to recover text-aligned features of the masked patches via
reasoning with the visible regions along the spatial and temporal dimensions,
which enhances the discriminativeness of local visual features and the
fine-grained cross-modality alignment. Our method outperforms state-of-the-art
methods for text-to-video retrieval on four datasets with both zero-shot and
fine-tune evaluation protocols. Our approach also surpasses the baseline models
significantly on zero-shot action recognition, which can be cast as
video-to-text retrieval.
