  This paper addresses task planning problems for language-instructed robot
teams. Tasks are expressed in natural language (NL), requiring the robots to
apply their capabilities at various locations and semantic objects. Several
recent works have addressed similar planning problems by leveraging pre-trained
Large Language Models (LLMs) to design effective multi-robot plans. However,
these approaches lack mission completion guarantees. To address this challenge,
we introduce a new decentralized LLM-based planner, called S-ATLAS for Safe
plAnning for Teams of Language-instructed AgentS, that is capable of achieving
user-defined mission success rates. This is accomplished by leveraging
conformal prediction (CP), a distribution-free uncertainty quantification tool
in black-box models. CP allows the proposed multi-robot planner to reason about
its inherent uncertainty in a decentralized fashion, enabling robots to make
individual decisions when they are sufficiently certain and seek help
otherwise. We show, both theoretically and empirically, that the proposed
planner can achieve user-specified task success rates while minimizing the
overall number of help requests. We provide comparative experiments against
related works showing that our method is significantly more computational
efficient and achieves lower help rates. The advantage of our algorithm over
baselines becomes more pronounced with increasing robot team size.
