  While intelligent virtual assistants like Siri, Alexa, and Google Assistant
have become ubiquitous in modern life, they still face limitations in their
ability to follow multi-step instructions and accomplish complex goals
articulated in natural language. However, recent breakthroughs in large
language models (LLMs) show promise for overcoming existing barriers by
enhancing natural language processing and reasoning capabilities. Though
promising, applying LLMs to create more advanced virtual assistants still faces
challenges like ensuring robust performance and handling variability in
real-world user commands. This paper proposes a novel LLM-based virtual
assistant that can automatically perform multi-step operations within mobile
apps based on high-level user requests. The system represents an advance in
assistants by providing an end-to-end solution for parsing instructions,
reasoning about goals, and executing actions. LLM-based Process Automation
(LLMPA) has modules for decomposing instructions, generating descriptions,
detecting interface elements, predicting next actions, and error checking.
Experiments demonstrate the system completing complex mobile operation tasks in
Alipay based on natural language instructions. This showcases how large
language models can enable automated assistants to accomplish real-world tasks.
The main contributions are the novel LLMPA architecture optimized for app
process automation, the methodology for applying LLMs to mobile apps, and
demonstrations of multi-step task completion in a real-world environment.
Notably, this work represents the first real-world deployment and extensive
evaluation of a large language model-based virtual assistant in a widely used
mobile application with an enormous user base numbering in the hundreds of
millions.
