  Entity disambiguation (ED) is the last step of entity linking (EL), when
candidate entities are reranked according to the context they appear in. All
datasets for training and evaluating models for EL consist of convenience
samples, such as news articles and tweets, that propagate the prior probability
bias of the entity distribution towards more frequently occurring entities. It
was previously shown that the performance of the EL systems on such datasets is
overestimated since it is possible to obtain higher accuracy scores by merely
learning the prior. To provide a more adequate evaluation benchmark, we
introduce the ShadowLink dataset, which includes 16K short text snippets
annotated with entity mentions. We evaluate and report the performance of
popular EL systems on the ShadowLink benchmark. The results show a considerable
difference in accuracy between more and less common entities for all of the EL
systems under evaluation, demonstrating the effects of prior probability bias
and entity overshadowing.
