  The Entity Set Expansion (ESE) task aims to expand a handful of seed entities
with new entities belonging to the same semantic class. Conventional ESE
methods are based on mono-modality (i.e., literal modality), which struggle to
deal with complex entities in the real world such as: (1) Negative entities
with fine-grained semantic differences. (2) Synonymous entities. (3) Polysemous
entities. (4) Long-tailed entities. These challenges prompt us to propose
Multi-modal Entity Set Expansion (MESE), where models integrate information
from multiple modalities to represent entities. Intuitively, the benefits of
multi-modal information for ESE are threefold: (1) Different modalities can
provide complementary information. (2) Multi-modal information provides a
unified signal via common visual properties for the same semantic class or
entity. (3) Multi-modal information offers robust alignment signal for
synonymous entities. To assess the performance of model in MESE and facilitate
further research, we constructed the MESED dataset which is the first
multi-modal dataset for ESE with large-scale and elaborate manual calibration.
A powerful multi-modal model MultiExpan is proposed which is pre-trained on
four multimodal pre-training tasks. The extensive experiments and analyses on
MESED demonstrate the high quality of the dataset and the effectiveness of our
MultiExpan, as well as pointing the direction for future research.
