  Handwritten text recognition has been widely studied in the last decades for
its numerous applications. Nowadays, the state-of-the-art approach consists in
a three-step process. The document is segmented into text lines, which are then
ordered and recognized. However, this three-step approach has many drawbacks.
The three steps are treated independently whereas they are closely related.
Errors accumulate from one step to the other. The ordering step is based on
heuristic rules which prevent its use for documents with a complex layouts or
for heterogeneous documents. The need for additional physical segmentation
annotations for training the segmentation stage is inherent to this approach.
In this thesis, we propose to tackle these issues by performing the handwritten
text recognition of whole document in an end-to-end way. To this aim, we
gradually increase the difficulty of the recognition task, moving from isolated
lines to paragraphs, and then to whole documents. We proposed an approach at
the line level, based on a fully convolutional network, in order to design a
first generic feature extraction step for the handwriting recognition task.
Based on this preliminary work, we studied two different approaches to
recognize handwritten paragraphs. We reached state-of-the-art results at
paragraph level on the RIMES 2011, IAM and READ 2016 datasets and outperformed
the line-level state of the art on these datasets. We finally proposed the
first end-to-end approach dedicated to the recognition of both text and layout,
at document level. Characters and layout tokens are sequentially predicted
following a learned reading order. We proposed two new metrics we used to
evaluate this task on the RIMES 2009 and READ 2016 dataset, at page level and
double-page level.
