  The rapid advancement of large language models (LLMs) has led to a new era
marked by the development of autonomous applications in real-world scenarios,
which drives innovation in creating advanced web agents. Existing web agents
typically only handle one input modality and are evaluated only in simplified
web simulators or static web snapshots, greatly limiting their applicability in
real-world scenarios. To bridge this gap, we introduce WebVoyager, an
innovative Large Multimodal Model (LMM) powered web agent that can complete
user instructions end-to-end by interacting with real-world websites. Moreover,
we establish a new benchmark by compiling real-world tasks from 15 popular
websites and introduce an automatic evaluation protocol leveraging multimodal
understanding abilities of GPT-4V to evaluate open-ended web agents. We show
that WebVoyager achieves a 59.1% task success rate on our benchmark,
significantly surpassing the performance of both GPT-4 (All Tools) and the
WebVoyager (text-only) setups, underscoring the exceptional capability of
WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement
with human judgment, indicating its effectiveness in providing reliable and
accurate assessments of web agents.
