  The advancement of large language models (LLMs) relies on evaluation using
public benchmarks, but data contamination can lead to overestimated
performance. Previous researches focus on detecting contamination by
determining whether the model has seen the exact same data during training. In
this work, we argue that even training on data similar to benchmark data
inflates performance on in-distribution tasks without improving overall
capacity, which we called In-distribution contamination. To effectively detect
in-distribution contamination, we propose DICE, a novel method that leverages
the internal states of LLMs to locate-then-detect the contamination. DICE first
identifies the most sensitive layer to contamination, then trains a classifier
based on the internal states of that layer. Experiments reveal DICE's high
accuracy in detecting in-distribution contamination across various LLMs and
math reasoning datasets. We also show the generalization capability of the
trained DICE detector, which is able to detect contamination across multiple
benchmarks with similar distributions. Additionally, we find that the DICE
detection scores are positively correlated with the performance of ten LLMs
fine-tuned by either us or other organizations on four math reasoning datasets
(with $R^2$ values between 0.6 and 0.75). This indicates that the
in-distribution contamination problem potentially lead to an overestimation of
the true capabilities of many existing models. The code and data are available
at https://github.com/THU-KEG/DICE.
