  In the field of robotics and computer vision, efficient and accurate semantic
mapping remains a significant challenge due to the growing demand for
intelligent machines that can comprehend and interact with complex
environments. Conventional panoptic mapping methods, however, are limited by
predefined semantic classes, thus making them ineffective for handling novel or
unforeseen objects. In response to this limitation, we introduce the Unified
Promptable Panoptic Mapping (UPPM) method. UPPM utilizes recent advances in
foundation models to enable real-time, on-demand label generation using natural
language prompts. By incorporating a dynamic labeling strategy into traditional
panoptic mapping techniques, UPPM provides significant improvements in
adaptability and versatility while maintaining high performance levels in map
reconstruction. We demonstrate our approach on real-world and simulated
datasets. Results show that UPPM can accurately reconstruct scenes and segment
objects while generating rich semantic labels through natural language
interactions. A series of ablation experiments validated the advantages of
foundation model-based labeling over fixed label sets.
