  Self Supervised Learning (SSL) has emerged as a powerful paradigm to tackle
data landscapes with absence of human supervision. The ability to learn
meaningful tasks without the use of labeled data makes SSL a popular method to
manage large chunks of data in the absence of labels. However, recent work
indicates SSL to be vulnerable to backdoor attacks, wherein models can be
controlled, possibly maliciously, to suit an adversary's motives. Li et. al
(2022) introduce a novel frequency-based backdoor attack: CTRL. They show that
CTRL can be used to efficiently and stealthily gain control over a victim's
model trained using SSL. In this work, we devise two defense strategies against
frequency-based attacks in SSL: One applicable before model training and the
second to be applied during model inference. Our first contribution utilizes
the invariance property of the downstream task to defend against backdoor
attacks in a generalizable fashion. We observe the ASR (Attack Success Rate) to
reduce by over 60% across experiments. Our Inference-time defense relies on
evasiveness of the attack and uses the luminance channel to defend against
attacks. Using object classification as the downstream task for SSL, we
demonstrate successful defense strategies that do not require re-training of
the model. Code is available at https://github.com/Aryan-Satpathy/Backdoor.
