  In this paper, we propose a Text-Degradation Invariant Auto Encoder
(Text-DIAE), a self-supervised model designed to tackle two tasks, text
recognition (handwritten or scene-text) and document image enhancement. We
start by employing a transformer-based architecture that incorporates three
pretext tasks as learning objectives to be optimized during pre-training
without the usage of labeled data. Each of the pretext objectives is
specifically tailored for the final downstream tasks. We conduct several
ablation experiments that confirm the design choice of the selected pretext
tasks. Importantly, the proposed model does not exhibit limitations of previous
state-of-the-art methods based on contrastive losses, while at the same time
requiring substantially fewer data samples to converge. Finally, we demonstrate
that our method surpasses the state-of-the-art in existing supervised and
self-supervised settings in handwritten and scene text recognition and document
image enhancement. Our code and trained models will be made publicly available
at~\url{ http://Upon_Acceptance}.
