  Large language model (LLM)-based mobile agents are increasingly popular due
to their capability to interact directly with mobile phone Graphic User
Interfaces (GUIs) and their potential to autonomously manage daily tasks.
Despite their promising prospects in both academic and industrial sectors,
little research has focused on benchmarking the performance of existing mobile
agents, due to the inexhaustible states of apps and the vague definition of
feasible action sequences. To address this challenge, we propose an efficient
and user-friendly benchmark, MobileAgentBench, designed to alleviate the burden
of extensive manual testing. We initially define 100 tasks across 10
open-source apps, categorized by multiple levels of difficulty. Subsequently,
we evaluate several existing mobile agents, including AppAgent and MobileAgent,
to thoroughly and systematically compare their performance. All materials are
accessible on our project webpage: https://MobileAgentBench.github.io,
contributing to the advancement of both academic and industrial fields.
