  Question-answering handwritten documents is a challenging task with numerous
real-world applications. This paper proposes a novel recognition-based approach
that improves upon the previous state-of-the-art on the HW-SQuAD and BenthamQA
datasets. Our model incorporates transformer-based document retrieval and
ensemble methods at the model level, achieving an Exact Match score of 82.02%
and 69% in HW-SQuAD and BenthamQA datasets, respectively, surpassing the
previous best recognition-based approach by 10.89% and 3%. We also enhance the
document retrieval component, boosting the top-5 retrieval accuracy from 90% to
95.30%. Our results demonstrate the significance of our proposed approach in
advancing question answering on handwritten documents. The code and trained
models will be publicly available to facilitate future research in this
critical area of natural language.
