  Vision-Language Pre-training (VLP) methods based on object detection enjoy
the rich knowledge of fine-grained object-text alignment but at the cost of
computationally expensive inference. Recent Visual-Transformer (ViT)-based
approaches circumvent this issue while struggling with long visual sequences
without detailed cross-modal alignment information. This paper introduces a
ViT-based VLP technique that efficiently incorporates object information
through a novel patch-text alignment mechanism. Specifically, we convert
object-level signals into patch-level ones and devise a Patch-Text Alignment
pre-training task (PTA) to learn a text-aware patch detector. By using
off-the-shelf delicate object annotations in 5\% training images, we jointly
train PTA with other conventional VLP objectives in an end-to-end manner,
bypassing the high computational cost of object detection and yielding an
effective patch detector that accurately detects text-relevant patches, thus
considerably reducing patch sequences and accelerating computation within the
ViT backbone. Our experiments on a variety of widely-used benchmarks reveal
that our method achieves a speedup of nearly 88\% compared to prior VLP models
while maintaining competitive or superior performance on downstream tasks with
similar model size and data scale.
