  Visual private information leakage is an emerging key issue for the fast
growing applications of video understanding like activity recognition. Existing
approaches for mitigating privacy leakage in action recognition require privacy
labels along with the action labels from the video dataset. However, annotating
frames of video dataset for privacy labels is not feasible. Recent developments
of self-supervised learning (SSL) have unleashed the untapped potential of the
unlabeled data. For the first time, we present a novel training framework which
removes privacy information from input video in a self-supervised manner
without requiring privacy labels. Our training framework consists of three main
components: anonymization function, self-supervised privacy removal branch, and
action recognition branch. We train our framework using a minimax optimization
strategy to minimize the action recognition cost function and maximize the
privacy cost function through a contrastive self-supervised loss. Employing
existing protocols of known-action and privacy attributes, our framework
achieves a competitive action-privacy trade-off to the existing
state-of-the-art supervised methods. In addition, we introduce a new protocol
to evaluate the generalization of learned the anonymization function to
novel-action and privacy attributes and show that our self-supervised framework
outperforms existing supervised methods. Code available at:
https://github.com/DAVEISHAN/SPAct
