  Scene text removal (STR) aims at replacing text strokes in natural scenes
with visually coherent backgrounds. Recent STR approaches rely on iterative
refinements or explicit text masks, resulting in high complexity and
sensitivity to the accuracy of text localization. Moreover, most existing STR
methods adopt convolutional architectures while the potential of vision
Transformers (ViTs) remains largely unexplored. In this paper, we propose a
simple-yet-effective ViT-based text eraser, dubbed ViTEraser. Following a
concise encoder-decoder framework, ViTEraser can easily incorporate various
ViTs to enhance long-range modeling. Specifically, the encoder hierarchically
maps the input image into the hidden space through ViT blocks and patch
embedding layers, while the decoder gradually upsamples the hidden features to
the text-erased image with ViT blocks and patch splitting layers. As ViTEraser
implicitly integrates text localization and inpainting, we propose a novel
end-to-end pretraining method, termed SegMIM, which focuses the encoder and
decoder on the text box segmentation and masked image modeling tasks,
respectively. Experimental results demonstrate that ViTEraser with SegMIM
achieves state-of-the-art performance on STR by a substantial margin and
exhibits strong generalization ability when extended to other tasks,
\textit{e.g.}, tampered scene text detection. Furthermore, we comprehensively
explore the architecture, pretraining, and scalability of the ViT-based
encoder-decoder for STR, which provides deep insights into the application of
ViT to the STR field. Code is available at
https://github.com/shannanyinxiang/ViTEraser.
