  Information Extraction processes in handwritten documents tend to rely on
obtaining an automatic transcription and performing Named Entity Recognition
(NER) over such transcription. For this reason, in publicly available datasets,
the performance of the systems is usually evaluated with metrics particular to
each dataset. Moreover, most of the metrics employed are sensitive to reading
order errors. Therefore, they do not reflect the expected final application of
the system and introduce biases in more complex documents. In this paper, we
propose and publicly release a set of reading order independent metrics
tailored to Information Extraction evaluation in handwritten documents. In our
experimentation, we perform an in-depth analysis of the behavior of the metrics
to recommend what we consider to be the minimal set of metrics to evaluate a
task correctly.
