  We propose the Data Contamination Quiz (DCQ), a simple and effective approach
to detect data contamination in large language models (LLMs) and estimate the
amount of it. Specifically, we frame data contamination detection as a series
of multiple-choice questions and devise a quiz format wherein three perturbed
versions of each subsampled instance from a specific dataset partition (e.g.,
GSM8k test set) are created. These changes only include word-level
perturbations. The generated perturbations, along with the original dataset
instance, form the options in the DCQ, with an extra option accommodating the
possibility of selecting none of the provided options. Given that the only
distinguishing signal among the options is the exact wording with respect to
the original dataset instance, an LLM, when tasked with identifying the
original dataset instance, gravitates towards selecting the original one if it
has been exposed to it in its pre-training phase -- a trait intrinsic to LLMs.
While accounting for positional biases in LLMs, the quiz performance reveals
the contamination level for the model being examined with the dataset partition
to which the quiz pertains. Applied to various datasets with GPT-4 and GPT-3.5,
our findings -- while fully lacking access to pre-training data and model
parameters -- suggest that DCQ achieves state-of-the-art results and uncovers
greater contamination/memorization levels compared to existing methods and
proficiently bypasses more safety filters, especially those set to avoid
generating copyrighted contents.
