  Various techniques have been proposed to leverage the capabilities of code
language models (CLMs) for SE tasks. While these techniques typically evaluate
their effectiveness using publicly available datasets, the evaluation can be
subject to data contamination threats where the evaluation datasets have
already been used to train the concerned CLMs. This can significantly affect
the reliability of the evaluation. Different countermeasures have been
suggested to mitigate the data contamination threat. Countermeasures include
using more recent data, curating new data, and refactoring existing data are
introduced, yet it is unclear whether these countermeasures could really
mitigate data contamination threats to model evaluation. To fill the gap, we
systematically study to quantify the impacts of these countermeasures on CLMs'
performance. To facilitate the study, we collected over 2 million Python
functions with timestamps ranging from January 1st, 2018, to December 31st,
2023. The data created before the models' cut-off date are considered
"contaminated data", while the data where the countermeasures are taken are
regarded as "cleansed data". We study the impact of these countermeasures by
investigating the difference in CLMs' performance on contaminated and cleansed
data derived from different countermeasures. Our experiments yield several
interesting observations. For instance, CLMs do not necessarily perform worse
on data after the models' cut-off date; on the contrary, they sometimes perform
better. In addition, refactoring did not always result in decreased
performance; it could lead to improvements instead. Furthermore, existing
metrics such as perplexity cannot distinguish contaminated/cleansed data. We
hope that the results and observations could help deepen the understanding of
CLMs' capabilities and inform the community about data contamination.
