  Pre-trained vision-language models~(VLMs) are the de-facto foundation models
for various downstream tasks. However, scene text recognition methods still
prefer backbones pre-trained on a single modality, namely, the visual modality,
despite the potential of VLMs to serve as powerful scene text readers. For
example, CLIP can robustly identify regular (horizontal) and irregular
(rotated, curved, blurred, or occluded) text in images. With such merits, we
transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet
effective STR method built upon image and text encoders of CLIP. It has two
encoder-decoder branches: a visual branch and a cross-modal branch. The visual
branch provides an initial prediction based on the visual feature, and the
cross-modal branch refines this prediction by addressing the discrepancy
between the visual feature and text semantics. To fully leverage the
capabilities of both branches, we design a dual predict-and-refine decoding
scheme for inference. We scale CLIP4STR in terms of the model size,
pre-training data, and training data, achieving state-of-the-art performance on
11 STR benchmarks. Additionally, a comprehensive empirical study is provided to
enhance the understanding of the adaptation of CLIP to STR. We believe our
method establishes a simple yet strong baseline for future STR research with
VLMs.
