  It is essential to detect functional differences in various software
engineering tasks, such as automated program repair, mutation testing, and code
refactoring. The problem of detecting functional differences between two
programs can be reduced to searching for a difference exposing test (DET): a
test input that results in different outputs on the subject programs. In this
paper, we propose Mokav, a novel execution-driven tool that leverages LLMs to
generate DETs. Mokav takes two versions of a program (P and Q) and an example
test input. When successful, Mokav generates a valid DET, a test input that
leads to different outputs on P and Q. Mokav iteratively prompts an LLM with a
specialized prompt to generate new test inputs. At each iteration, Mokav
provides execution-based feedback regarding previously generated tests until
the LLM produces a DET. We evaluate Mokav on 1,535 pairs of Python programs
collected from the Codeforces competition platform and 32 pairs of programs
from the QuixBugs dataset. Our experiments show that Mokav outperforms the
state-of-the-art, Pynguin and Differential Prompting, by a large margin. Mokav
can generate DETs for 81.7% (1,255/1,535) of the program pairs in our benchmark
(versus 4.9% for Pynguin and 37.3% for Differential Prompting). We demonstrate
that all components in our system, including the iterative and execution-driven
approaches, contribute to its high effectiveness.
