  Handwritten text recognition in low resource scenarios, such as manuscripts
with rare alphabets, is a challenging problem. The main difficulty comes from
the very few annotated data and the limited linguistic information (e.g.
dictionaries and language models). Thus, we propose a few-shot learning-based
handwriting recognition approach that significantly reduces the human labor
annotation process, requiring only few images of each alphabet symbol. The
method consists in detecting all the symbols of a given alphabet in a textline
image and decoding the obtained similarity scores to the final sequence of
transcribed symbols. Our model is first pretrained on synthetic line images
generated from any alphabet, even though different from the target domain. A
second training step is then applied to diminish the gap between the source and
target data. Since this retraining would require annotation of thousands of
handwritten symbols together with their bounding boxes, we propose to avoid
such human effort through an unsupervised progressive learning approach that
automatically assigns pseudo-labels to the non-annotated data. The evaluation
on different manuscript datasets show that our model can lead to competitive
results with a significant reduction in human effort. The code will be publicly
available in this repository: \url{https://github.com/dali92002/HTRbyMatching}
