  Named entity recognition (NER) is a fundamental component in the modern
language understanding pipeline. Public NER resources such as annotated data
and model services are available in many domains. However, given a particular
downstream application, there is often no single NER resource that supports all
the desired entity types, so users must leverage multiple resources with
different tag sets. This paper presents a marginal distillation (MARDI)
approach for training a unified NER model from resources with disjoint or
heterogeneous tag sets. In contrast to recent works, MARDI merely requires
access to pre-trained models rather than the original training datasets. This
flexibility makes it easier to work with sensitive domains like healthcare and
finance. Furthermore, our approach is general enough to integrate with
different NER architectures, including local models (e.g., BiLSTM) and global
models (e.g., CRF). Experiments on two benchmark datasets show that MARDI
performs on par with a strong marginal CRF baseline, while being more flexible
in the form of required NER resources. MARDI also sets a new state of the art
on the progressive NER task. MARDI significantly outperforms the
start-of-the-art model on the task of progressive NER.
