  Large Language Models (LLMs) have demonstrated strong natural language
processing and code synthesis capabilities, which has led to their rapid
adoption in software engineering applications. However, details about LLM
training data are often not made public, which has caused concern as to whether
existing bug benchmarks are included. In lieu of the training data for the
popular GPT models, we examine the training data of the open-source LLM
StarCoder, and find it likely that data from the widely used Defects4J
benchmark was included, raising the possibility of its inclusion in GPT
training data as well. This makes it difficult to tell how well LLM-based
results on Defects4J would generalize, as for any results it would be unclear
whether a technique's performance is due to LLM generalization or memorization.
To remedy this issue and facilitate continued research on LLM-based SE, we
present the GitHub Recent Bugs (GHRB) dataset, which includes 76 real-world
Java bugs that were gathered after the OpenAI data cut-off point.
