  In contrast to Connectionist Temporal Classification (CTC) approaches,
Sequence-To-Sequence (S2S) models for Handwritten Text Recognition (HTR) suffer
from errors such as skipped or repeated words which often occur at the end of a
sequence. In this paper, to combine the best of both approaches, we propose to
use the CTC-Prefix-Score during S2S decoding. Hereby, during beam search, paths
that are invalid according to the CTC confidence matrix are penalised. Our
network architecture is composed of a Convolutional Neural Network (CNN) as
visual backbone, bidirectional Long-Short-Term-Memory-Cells (LSTMs) as encoder,
and a decoder which is a Transformer with inserted mutual attention layers. The
CTC confidences are computed on the encoder while the Transformer is only used
for character-wise S2S decoding. We evaluate this setup on three HTR data sets:
IAM, Rimes, and StAZH. On IAM, we achieve a competitive Character Error Rate
(CER) of 2.95% when pretraining our model on synthetic data and including a
character-based language model for contemporary English. Compared to other
state-of-the-art approaches, our model requires about 10-20 times less
parameters. Access our shared implementations via this link to GitHub:
https://github.com/Planet-AI-GmbH/tfaip-hybrid-ctc-s2s.
