  The development of autonomous agents increasingly relies on Multimodal
Language Models (MLMs) to perform tasks described in natural language with GUI
environments, such as websites, desktop computers, or mobile phones. Existing
benchmarks for MLM agents in interactive environments are limited by their
focus on a single environment, lack of detailed and generalized evaluation
methods, and the complexities of constructing tasks and evaluators. To overcome
these limitations, we introduce Crab, the first agent benchmark framework
designed to support cross-environment tasks, incorporating a graph-based
fine-grained evaluation method and an efficient mechanism for task and
evaluator construction. Our framework supports multiple devices and can be
easily extended to any environment with a Python interface. Leveraging Crab, we
developed a cross-platform Crab Benchmark-v0 comprising 100 tasks in computer
desktop and mobile phone environments. We evaluated four advanced MLMs using
different single and multi-agent system configurations on this benchmark. The
experimental results demonstrate that the single agent with GPT-4o achieves the
best completion ratio of 35.26%. All framework code, agent code, and task
datasets are publicly available at https://github.com/camel-ai/crab.
