  Social media data is plagued by the redundancy problem caused by its noisy
nature, leading to increased training time and model bias. To address this
issue, we propose a novel approach called generative deduplication. It aims to
remove duplicate text from noisy social media data and mitigate model bias. By
doing so, it can improve social media language understanding performance and
save training time. Extensive experiments demonstrate that the proposed
generative deduplication can effectively reduce training samples while
improving performance. This evidence suggests the effectiveness of generative
deduplication and its importance in social media language understanding.
