  The common sense reasoning abilities and vast general knowledge of Large
Language Models (LLMs) make them a natural fit for interpreting user requests
in a Smart Home assistant context. LLMs, however, lack specific knowledge about
the user and their home limit their potential impact. SAGE (Smart Home Agent
with Grounded Execution), overcomes these and other limitations by using a
scheme in which a user request triggers an LLM-controlled sequence of discrete
actions. These actions can be used to retrieve information, interact with the
user, or manipulate device states. SAGE controls this process through a
dynamically constructed tree of LLM prompts, which help it decide which action
to take next, whether an action was successful, and when to terminate the
process. The SAGE action set augments an LLM's capabilities to support some of
the most critical requirements for a Smart Home assistant. These include:
flexible and scalable user preference management ("is my team playing
tonight?"), access to any smart device's full functionality without
device-specific code via API reading "turn down the screen brightness on my
dryer", persistent device state monitoring ("remind me to throw out the milk
when I open the fridge"), natural device references using only a photo of the
room ("turn on the light on the dresser"), and more. We introduce a benchmark
of 50 new and challenging smart home tasks where SAGE achieves a 75% success
rate, significantly outperforming existing LLM-enabled baselines (30% success
rate).
