  The digitisation of historical documents has provided historians with
unprecedented research opportunities. Yet, the conventional approach to
analysing historical documents involves converting them from images to text
using OCR, a process that overlooks the potential benefits of treating them as
images and introduces high levels of noise. To bridge this gap, we take
advantage of recent advancements in pixel-based language models trained to
reconstruct masked patches of pixels instead of predicting token distributions.
Due to the scarcity of real historical scans, we propose a novel method for
generating synthetic scans to resemble real historical documents. We then
pre-train our model, PHD, on a combination of synthetic scans and real
historical newspapers from the 1700-1900 period. Through our experiments, we
demonstrate that PHD exhibits high proficiency in reconstructing masked image
patches and provide evidence of our model's noteworthy language understanding
capabilities. Notably, we successfully apply our model to a historical QA task,
highlighting its usefulness in this domain.
