  Grounded Multimodal Named Entity Recognition (GMNER) task aims to identify
named entities, entity types and their corresponding visual regions. GMNER task
exhibits two challenging attributes: 1) The tenuous correlation between images
and text on social media contributes to a notable proportion of named entities
being ungroundable. 2) There exists a distinction between coarse-grained noun
phrases used in similar tasks (e.g., phrase localization) and fine-grained
named entities. In this paper, we propose RiVEG, a unified framework that
reformulates GMNER into a joint MNER-VE-VG task by leveraging large language
models (LLMs) as connecting bridges. This reformulation brings two benefits: 1)
It enables us to optimize the MNER module for optimal MNER performance and
eliminates the need to pre-extract region features using object detection
methods, thus naturally addressing the two major limitations of existing GMNER
methods. 2) The introduction of Entity Expansion Expression module and Visual
Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG).
This endows the proposed framework with unlimited data and model scalability.
Furthermore, to address the potential ambiguity stemming from the
coarse-grained bounding box output in GMNER, we further construct the new
Segmented Multimodal Named Entity Recognition (SMNER) task and corresponding
Twitter-SMNER dataset aimed at generating fine-grained segmentation masks, and
experimentally demonstrate the feasibility and effectiveness of using box
prompt-based Segment Anything Model (SAM) to empower any GMNER model with the
ability to accomplish the SMNER task. Extensive experiments demonstrate that
RiVEG significantly outperforms SoTA methods on four datasets across the MNER,
GMNER, and SMNER tasks.
