  Empirical methods in geoparsing have thus far lacked a standard evaluation
framework describing the task, metrics and data used to compare
state-of-the-art systems. Evaluation is further made inconsistent, even
unrepresentative of real-world usage by the lack of distinction between the
different types of toponyms, which necessitates new guidelines, a consolidation
of metrics and a detailed toponym taxonomy with implications for Named Entity
Recognition (NER) and beyond. To address these deficiencies, our manuscript
introduces a new framework in three parts. Part 1) Task Definition: clarified
via corpus linguistic analysis proposing a fine-grained Pragmatic Taxonomy of
Toponyms. Part 2) Metrics: discussed and reviewed for a rigorous evaluation
including recommendations for NER/Geoparsing practitioners. Part 3) Evaluation
Data: shared via a new dataset called GeoWebNews to provide test/train examples
and enable immediate use of our contributions. In addition to fine-grained
Geotagging and Toponym Resolution (Geocoding), this dataset is also suitable
for prototyping and evaluating machine learning NLP models.
