  With the advancement of software rendering techniques, GUI pages in mobile
apps now encompass a wealth of visual information, where the visual semantics
of each page contribute to the overall app logic, presenting new challenges to
software testing. Despite the progress in automated Graphical User Interface
(GUI) testing, the absence of testing oracles has constrained its efficacy to
identify only crash bugs with evident abnormal signals. Nonetheless, there are
still a considerable number of non-crash bugs, ranging from unexpected
behaviors to misalignments, often evading detection by existing techniques.
While these bugs can exhibit visual cues that serve as potential testing
oracles, they often entail a sequence of screenshots, and detecting them
necessitates an understanding of the operational logic among GUI page
transitions, which is challenging traditional techniques. Considering the
remarkable performance of Multimodal Large Language Models (MLLM) in visual and
language understanding, this paper proposes a vision-driven automated GUI
testing approach VisionDroid to detect non-crash functional bugs with MLLM. It
begins by extracting GUI text information and aligning it with screenshots to
form a vision prompt, enabling MLLM to understand GUI context. The
function-aware explorer then employs MLLM for deeper and function-oriented GUI
page exploration, while the logic-aware bug detector segments the entire
exploration history into logically cohesive parts and prompts the MLLM for bug
detection. We evaluate VisionDroid on three datasets and compare it with 10
baselines, demonstrating its excellent performance. The ablation study further
proves the contribution of each module. Moreover, VisionDroid identifies 29 new
bugs on Google Play, of which 19 have been confirmed and fixed.
