  Language agents, built on top of language models (LMs), are systems that can
interact with complex environments, such as the open web. In this work, we
examine whether such agents can perform realistic and time-consuming tasks on
the web, e.g., monitoring real-estate markets or locating relevant nearby
businesses. We introduce AssistantBench, a challenging new benchmark consisting
of 214 realistic tasks that can be automatically evaluated, covering different
scenarios and domains. We find that AssistantBench exposes the limitations of
current systems, including language models and retrieval-augmented language
models, as no model reaches an accuracy of more than 25 points. While
closed-book LMs perform well, they exhibit low precision since they tend to
hallucinate facts. State-of-the-art web agents reach a score of near zero.
Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
outperforms previous agents, and an ensemble of SPA and closed-book models
reaches the best overall performance. Moreover, we analyze failures of current
systems and highlight that web navigation remains a major challenge.
