  In Self-Supervised Learning (SSL), it is known that frequent occurrences of
the collision in which target data and its negative samples share the same
class can decrease performance. Especially in real-world data such as crawled
data or robot-gathered observations, collisions may occur more often due to the
duplicates in the data. To deal with this problem, we claim that sampling
negative samples from the adaptively debiased distribution in the memory makes
the model more stable than sampling from a biased dataset directly. In this
paper, we introduce a novel SSL framework with adaptive Duplicate Elimination
(DUEL) inspired by the human working memory. The proposed framework
successfully prevents the downstream task performance from degradation due to a
dramatic inter-class imbalance.
