  The surge in popularity of Large Language Models (LLMs) has opened doors for
new approaches to the creation of interactive agents. However, managing the
temporal behavior of such agents over the course of an interaction remains
challenging. The stateful, long-term horizon and quantitative reasoning
required for coherent agent behavior does not fit well into the LLM paradigm.
We propose a combination of formal logic-based program synthesis and LLM
content generation to create generative agents that adhere to temporal
constraints. Our approach uses Temporal Stream Logic (TSL) to generate an
automaton that enforces a temporal structure on an agent and leaves the details
of each action for a moment in time to an LLM. By using TSL, we are able to
augment the generative agent where users have a higher level of guarantees on
behavior, better interpretability of the system, and more ability to build
agents in a modular way. We evaluate our approach on different tasks involved
in creating a coherent interactive agent specialized for various application
domains. We found that over all of the tasks, our approach using TSL achieves
at least 96% adherence, whereas the pure LLM-based approach demonstrates as low
as 14.67% adherence.
