  Many automated test generation techniques have been developed to aid
developers with writing tests. To facilitate full automation, most existing
techniques aim to either increase coverage, or generate exploratory inputs.
However, existing test generation techniques largely fall short of achieving
more semantic objectives, such as generating tests to reproduce a given bug
report. Reproducing bugs is nonetheless important, as our empirical study shows
that the number of tests added in open source repositories due to issues was
about 28% of the corresponding project test suite size. Meanwhile, due to the
difficulties of transforming the expected program semantics in bug reports into
test oracles, existing failure reproduction techniques tend to deal exclusively
with program crashes, a small subset of all bug reports. To automate test
generation from general bug reports, we propose LIBRO, a framework that uses
Large Language Models (LLMs), which have been shown to be capable of performing
code-related tasks. Since LLMs themselves cannot execute the target buggy code,
we focus on post-processing steps that help us discern when LLMs are effective,
and rank the produced tests according to their validity. Our evaluation of
LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate
failure reproducing test cases for 33% of all studied cases (251 out of 750),
while suggesting a bug reproducing test in first place for 149 bugs. To
mitigate data contamination, we also evaluate LIBRO against 31 bug reports
submitted after the collection of the LLM training data terminated: LIBRO
produces bug reproducing tests for 32% of the studied bug reports. Overall, our
results show LIBRO has the potential to significantly enhance developer
efficiency by automatically generating tests from bug reports.
