  In past years, we have been dedicated to automating user acceptance testing
(UAT) process of WeChat Pay, one of the most influential mobile payment
applications in China. A system titled XUAT has been developed for this
purpose. However, there is still a human-labor-intensive stage, i.e, test
scripts generation, in the current system. Therefore, in this paper, we
concentrate on methods of boosting the automation level of the current system,
particularly the stage of test scripts generation. With recent notable
successes, large language models (LLMs) demonstrate significant potential in
attaining human-like intelligence and there has been a growing research area
that employs LLMs as autonomous agents to obtain human-like decision-making
capabilities. Inspired by these works, we propose an LLM-powered multi-agent
collaborative system, named XUAT-Copilot, for automated UAT. The proposed
system mainly consists of three LLM-based agents responsible for action
planning, state checking and parameter selecting, respectively, and two
additional modules for state sensing and case rewriting. The agents interact
with testing device, make human-like decision and generate action command in a
collaborative way. The proposed multi-agent system achieves a close
effectiveness to human testers in our experimental studies and gains a
significant improvement of Pass@1 accuracy compared with single-agent
architecture. More importantly, the proposed system has launched in the formal
testing environment of WeChat Pay mobile app, which saves a considerable amount
of manpower in the daily development work.
