  For web agents to be practically useful, they must adapt to the continuously
evolving web environment characterized by frequent updates to user interfaces
and content. However, most existing benchmarks only capture the static aspects
of the web. To bridge this gap, we introduce WebCanvas, an innovative online
evaluation framework for web agents that effectively addresses the dynamic
nature of web interactions. WebCanvas contains three main components to
facilitate realistic assessments: (1) A novel evaluation metric which reliably
capture critical intermediate actions or states necessary for task completions
while disregarding noise caused by insignificant events or changed
web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version
of original Mind2Web static dataset containing 542 tasks with 2439 intermediate
evaluation states; (3) Lightweight and generalizable annotation tools and
testing pipelines that enables the community to collect and maintain the
high-quality, up-to-date dataset. Building on WebCanvas, we open-source an
agent framework with extensible modules for reasoning, providing a foundation
for the community to conduct online inference and evaluations. Our
best-performing agent achieves a task success rate of 23.1% and a task
completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we
analyze the performance discrepancies across various websites, domains, and
experimental environments. We encourage the community to contribute further
insights on online agent evaluation, thereby advancing this field of research.
