  We introduce Semantic Parsing in Contextual Environments (SPICE), a task
designed to enhance artificial agents' contextual awareness by integrating
multimodal inputs with prior contexts. SPICE goes beyond traditional semantic
parsing by offering a structured, interpretable framework for dynamically
updating an agent's knowledge with new information, mirroring the complexity of
human communication. We develop the VG-SPICE dataset, crafted to challenge
agents with visual scene graph construction from spoken conversational
exchanges, highlighting speech and visual data integration. We also present the
Audio-Vision Dialogue Scene Parser (AViD-SP) developed for use on VG-SPICE.
These innovations aim to improve multimodal information processing and
integration. Both the VG-SPICE dataset and the AViD-SP model are publicly
available.
