  Autonomous agents that execute human tasks by controlling computers can
enhance human productivity and application accessibility. However, progress in
this field will be driven by realistic and reproducible benchmarks. We present
AndroidWorld, a fully functional Android environment that provides reward
signals for 116 programmatic tasks across 20 real-world Android apps. Unlike
existing interactive environments, which provide a static test set,
AndroidWorld dynamically constructs tasks that are parameterized and expressed
in natural language in unlimited ways, thus enabling testing on a much larger
and more realistic suite of tasks. Reward signals are derived from the
computer's system state, making them durable across task variations and
extensible across different apps. To demonstrate AndroidWorld's benefits and
mode of operation, we introduce a new computer control agent, M3A. M3A can
complete 30.6% of the AndroidWorld's tasks, leaving ample room for future work.
Furthermore, we adapt a popular desktop web agent to work on Android, which we
find to be less effective on mobile, suggesting future research is needed to
achieve universal, cross-domain agents. Finally, we conduct a robustness
analysis by testing M3A against a range of task variations on a representative
subset of tasks, demonstrating that variations in task parameters can
significantly alter a task's complexity and, consequently, an agent's
performance, highlighting the importance of testing agents under diverse
conditions. AndroidWorld and the experiments in this paper are available at
https://github.com/google-research/android_world.
