  Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal
task that aims to identify named entities, entity types and their corresponding
visual regions. GMNER task exhibits two challenging properties: 1) The weak
correlation between image-text pairs in social media results in a significant
portion of named entities being ungroundable. 2) There exists a distinction
between coarse-grained referring expressions commonly used in similar tasks
(e.g., phrase localization, referring expression comprehension) and
fine-grained named entities. In this paper, we propose RiVEG, a unified
framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging
large language models (LLMs) as a connecting bridge. This reformulation brings
two benefits: 1) It maintains the optimal MNER performance and eliminates the
need for employing object detection methods to pre-extract regional features,
thereby naturally addressing two major limitations of existing GMNER methods.
2) The introduction of entity expansion expression and Visual Entailment (VE)
module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables
RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding
capabilities of any current or prospective multimodal pretraining models.
Extensive experiments demonstrate that RiVEG outperforms state-of-the-art
methods on the existing GMNER dataset and achieves absolute leads of 10.65%,
6.21%, and 8.83% in all three subtasks.
