  Automatically detecting software failures is an important task and a
longstanding challenge. It requires finding failure-inducing test cases whose
test input can trigger the software's fault, and constructing an automated
oracle to detect the software's incorrect behaviors. Recent advancement of
large language models (LLMs) motivates us to study how far this challenge can
be addressed by ChatGPT, a state-of-the-art LLM. Unfortunately, our study shows
that ChatGPT has a low probability (28.8%) of finding correct failure-inducing
test cases for buggy programs. A possible reason is that finding
failure-inducing test cases requires analyzing the subtle code differences
between a buggy program and its correct version. When these two versions have
similar syntax, ChatGPT is weak at recognizing subtle code differences. Our
insight is that ChatGPT's performance can be substantially enhanced when
ChatGPT is guided to focus on the subtle code difference. We have an
interesting observation that ChatGPT is effective in inferring the intended
behaviors of a buggy program. The intended behavior can be leveraged to
synthesize programs, in order to make the subtle code difference between a
buggy program and its correct version (i.e., the synthesized program) explicit.
Driven by this observation, we propose a novel approach that synergistically
combines ChatGPT and differential testing to find failure-inducing test cases.
We evaluate our approach on Quixbugs (a benchmark of buggy programs), and
compare it with state-of-the-art baselines, including direct use of ChatGPT and
Pynguin. The experimental result shows that our approach has a much higher
probability (77.8%) of finding correct failure-inducing test cases, 2.7X as the
best baseline.
