  Large language models pretrained on extensive web corpora demonstrate
remarkable performance across a wide range of downstream tasks. However, a
growing concern is data contamination, where evaluation datasets may be
contained in the pretraining corpus, inflating model performance.
Decontamination, the process of detecting and removing such data, is a
potential solution; yet these contaminants may originate from altered versions
of the test set, evading detection during decontamination. How different types
of contamination impact the performance of language models on downstream tasks
is not fully understood. We present a taxonomy that categorizes the various
types of contamination encountered by LLMs during the pretraining phase and
identify which types pose the highest risk. We analyze the impact of
contamination on two key NLP tasks -- summarization and question answering --
revealing how different types of contamination influence task performance
during evaluation.
