  Recent advancements in Optical Character Recognition (OCR) have been driven
by transformer-based models. OCR systems are critical in numerous high-stakes
domains, yet their vulnerability to adversarial attack remains largely
uncharted territory, raising concerns about security and compliance with
emerging AI regulations. In this work we present a novel framework to assess
the resilience of Transformer-based OCR (TrOCR) models. We develop and assess
algorithms for both targeted and untargeted attacks. For the untargeted case,
we measure the Character Error Rate (CER), while for the targeted case we use
the success ratio. We find that TrOCR is highly vulnerable to untargeted
attacks and somewhat less vulnerable to targeted attacks. On a benchmark
handwriting data set, untargeted attacks can cause a CER of more than 1 without
being noticeable to the eye. With a similar perturbation size, targeted attacks
can lead to success rates of around $25\%$ -- here we attacked single tokens,
requiring TrOCR to output the tenth most likely token from a large vocabulary.
