  We highlight several issues in the evaluation of historical text
normalization systems that make it hard to tell how well these systems would
actually work in practice---i.e., for new datasets or languages; in comparison
to more na\"ive systems; or as a preprocessing step for downstream NLP tools.
We illustrate these issues and exemplify our proposed evaluation practices by
comparing two neural models against a na\"ive baseline system. We show that the
neural models generalize well to unseen words in tests on five languages;
nevertheless, they provide no clear benefit over the na\"ive baseline for
downstream POS tagging of an English historical collection. We conclude that
future work should include more rigorous evaluation, including both intrinsic
and extrinsic measures where possible.
