  Many popular programming languages, including C#, Java, and Python, support
exceptions. Exceptions are thrown during program execution if an unwanted event
happens, e.g., a method is invoked with an illegal argument value. Software
developers write exceptional behavior tests (EBTs) to check that their code
detects unwanted events and throws appropriate exceptions. Prior research
studies have shown the importance of EBTs, but those studies also highlighted
that developers put most of their efforts on "happy paths", e.g., paths without
unwanted events. To help developers fill the gap, we present the first
framework, dubbed exLong, that automatically generates EBTs. exLong is a large
language model instruction-tuned from CodeLlama and embeds reasoning about
traces that lead to throw statements, conditional expressions that guard throw
statements, and non-exceptional behavior tests that execute similar traces. We
compare exLong with the state-of-the-art models for test generation (CAT-LM)
and one of the strongest foundation models (GPT3.5), as well as with
analysis-based tools for test generation (Randoop and EvoSuite). Our results
show that exLong outperforms existing models and tools. Furthermore, we
contributed several pull requests to open-source projects and 23 EBTs generated
by exLong were already accepted.
