  Test suites tend to grow when software evolves, making it often infeasible to
execute all test cases with the allocated testing budgets, especially for large
software systems. Test suite minimization (TSM) is employed to improve the
efficiency of software testing by removing redundant test cases, thus reducing
testing time and resources, while maintaining the fault detection capability of
the test suite. Most existing TSM approaches rely on code coverage (white-box)
or model-based features, which are not always available to test engineers.
Recent TSM approaches that rely only on test code (black-box) have been
proposed, such as ATM and FAST-R. To address the scalability, we propose LTM
(Language model-based Test suite Minimization), a novel, scalable, and
black-box similarity-based TSM approach based on large language models (LLMs),
which is the first application of LLMs in the context of TSM. To support
similarity measurement for test code embeddings, we investigate five
pre-trained language models: CodeBERT, GraphCodeBERT, UniXcoder, StarEncoder,
and CodeLlama, on which we compute two similarity measures: Cosine Similarity
and Euclidean Distance. Our goal is to find similarity measures that are not
only computationally more efficient but can also better guide a Genetic
Algorithm (GA) to search for optimal minimized test suites, thus reducing the
overall search time. Experimental results show that the best configuration of
LTM (UniXcoder/Cosine) outperforms ATM in three aspects: (a) achieving a
slightly greater saving rate of testing time (41.72% versus 40.29%, on
average); (b) attaining a significantly higher fault detection rate (0.84
versus 0.81, on average); and, most importantly, (c) minimizing test suites
nearly five times faster on average, with higher gains for larger test suites
and systems, thus achieving much higher scalability.
