  Mobile apps are indispensable for people's daily life, and automated GUI
(Graphical User Interface) testing is widely used for app quality assurance.
There is a growing interest in using learning-based techniques for automated
GUI testing which aims at generating human-like actions and interactions.
However, the limitations such as low testing coverage, weak generalization, and
heavy reliance on training data, make an urgent need for a more effective
approach to generate human-like actions to thoroughly test mobile apps.
Inspired by the success of the Large Language Model (LLM), e.g., GPT-3 and
ChatGPT, in natural language understanding and question answering, we formulate
the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM
to chat with the mobile apps by passing the GUI page information to LLM to
elicit testing scripts, and executing them to keep passing the app feedback to
LLM, iterating the whole process. Within it, we extract the static context of
the GUI page and the dynamic context of the iterative testing process, design
prompts for inputting this information to LLM, and develop a neural matching
network to decode the LLM's output into actionable steps to execute the app. We
evaluate GPTDroid on 86 apps from Google Play, and its activity coverage is
71%, with 32% higher than the best baseline, and can detect 36% more bugs with
faster speed than the best baseline. GPTDroid also detects 48 new bugs on the
Google Play with 25 of them being confirmed/fixed. We further summarize the
capabilities of GPTDroid behind the superior performance, including semantic
text input, compound action, long meaningful test trace, and test case
prioritization.
