  Smart home assistants function best when user commands are direct and
well-specified (e.g., "turn on the kitchen light"), or when a hard-coded
routine specifies the response. In more natural communication, however, human
speech is unconstrained, often describing goals (e.g., "make it cozy in here"
or "help me save energy") rather than indicating specific target devices and
actions to take on those devices. Current systems fail to understand these
under-specified commands since they cannot reason about devices and settings as
they relate to human situations. We introduce large language models (LLMs) to
this problem space, exploring their use for controlling devices and creating
automation routines in response to under-specified user commands in smart
homes. We empirically study the baseline quality and failure modes of
LLM-created action plans with a survey of age-diverse users. We find that LLMs
can reason creatively to achieve challenging goals, but they experience
patterns of failure that diminish their usefulness. We address these gaps with
Sasha, a smarter smart home assistant. Sasha responds to loosely-constrained
commands like "make it cozy" or "help me sleep better" by executing plans to
achieve user goals, e.g., setting a mood with available devices, or devising
automation routines. We implement and evaluate Sasha in a hands-on user study,
showing the capabilities and limitations of LLM-driven smart homes when faced
with unconstrained user-generated scenarios.
