  Language agents have shown some ability to interact with an external
environment, e.g., a virtual world such as ScienceWorld, to perform complex
tasks, e.g., growing a plant, without the startup costs of reinforcement
learning. However, despite their zero-shot capabilities, these agents to date
do not continually improve over time beyond performance refinement on a
specific task. Here we present CLIN, the first language-based agent to achieve
this, so that it continually improves over multiple trials, including when both
the environment and task are varied, and without requiring parameter updates.
Our approach is to use a persistent, dynamic, textual memory centered on causal
abstractions (rather than general "helpful hints") that is regularly updated
after each trial so that the agent gradually learns useful knowledge for new
trials. In the ScienceWorld benchmark, CLIN is able to continually improve on
repeated trials on the same task and environment, outperforming
state-of-the-art reflective language agents like Reflexion by 23 absolute
points. CLIN can also transfer its learning to new environments (or new tasks),
improving its zero-shot performance by 4 points (13 for new tasks) and can
further improve performance there through continual memory updates, enhancing
performance by an additional 17 points (7 for new tasks). This suggests a new
architecture for agents built on frozen models that can still continually and
rapidly improve over time.
