  Real-world data usually exhibits a long-tailed distribution,with a few
frequent labels and a lot of few-shot labels. The study of institution name
normalization is a perfect application case showing this phenomenon. There are
many institutions worldwide with enormous variations of their names in the
publicly available literature. In this work, we first collect a large-scale
institution name normalization dataset LoT-insts1, which contains over 25k
classes that exhibit a naturally long-tailed distribution. In order to isolate
the few-shot and zero-shot learning scenarios from the massive many-shot
classes, we construct our test set from four different subsets: many-, medium-,
and few-shot sets, as well as a zero-shot open set. We also replicate several
important baseline methods on our data, covering a wide range from search-based
methods to neural network methods that use the pretrained BERT model. Further,
we propose our specially pretrained, BERT-based model that shows better
out-of-distribution generalization on few-shot and zero-shot test sets.
Compared to other datasets focusing on the long-tailed phenomenon, our dataset
has one order of magnitude more training data than the largest existing
long-tailed datasets and is naturally long-tailed rather than manually
synthesized. We believe it provides an important and different scenario to
study this problem. To our best knowledge, this is the first natural language
dataset that focuses on long-tailed and open-set classification problems.
