  In embodied instruction-following (EIF), the integration of pretrained
language models (LMs) as task planners emerges as a significant branch, where
tasks are planned at the skill level by prompting LMs with pretrained skills
and user instructions. However, grounding these pretrained skills in different
domains remains challenging due to their intricate entanglement with the
domain-specific knowledge. To address this challenge, we present a semantic
skill grounding (SemGro) framework that leverages the hierarchical nature of
semantic skills. SemGro recognizes the broad spectrum of these skills, ranging
from short-horizon low-semantic skills that are universally applicable across
domains to long-horizon rich-semantic skills that are highly specialized and
tailored for particular domains. The framework employs an iterative skill
decomposition approach, starting from the higher levels of semantic skill
hierarchy and then moving downwards, so as to ground each planned skill to an
executable level within the target domain. To do so, we use the reasoning
capabilities of LMs for composing and decomposing semantic skills, as well as
their multi-modal extension for assessing the skill feasibility in the target
domain. Our experiments in the VirtualHome benchmark show the efficacy of
SemGro in 300 cross-domain EIF scenarios.
