  Synthetic data has been proposed as a solution to address the issue of
high-quality data scarcity in the training of large language models (LLMs).
Studies have shown that synthetic data can effectively improve the performance
of LLMs on downstream benchmarks. However, despite its potential benefits, our
analysis suggests that there may be inherent flaws in synthetic data. The
uniform format of synthetic data can lead to pattern overfitting and cause
significant shifts in the output distribution, thereby reducing the model's
instruction-following capabilities. Our work delves into these specific flaws
associated with question-answer (Q-A) pairs, a prevalent type of synthetic
data, and presents a method based on unlearning techniques to mitigate these
flaws. The empirical results demonstrate the effectiveness of our approach,
which can reverse the instruction-following issues caused by pattern
overfitting without compromising performance on benchmarks at relatively low
cost. Our work has yielded key insights into the effective use of synthetic
data, aiming to promote more robust and efficient LLM training.
