  Classifiers in supervised learning have various security and privacy issues,
e.g., 1) data poisoning attacks, backdoor attacks, and adversarial examples on
the security side as well as 2) inference attacks and the right to be forgotten
for the training data on the privacy side. Various secure and
privacy-preserving supervised learning algorithms with formal guarantees have
been proposed to address these issues. However, they suffer from various
limitations such as accuracy loss, small certified security guarantees, and/or
inefficiency. Self-supervised learning is an emerging technique to pre-train
encoders using unlabeled data. Given a pre-trained encoder as a feature
extractor, supervised learning can train a simple yet accurate classifier using
a small amount of labeled training data. In this work, we perform the first
systematic, principled measurement study to understand whether and when a
pre-trained encoder can address the limitations of secure or privacy-preserving
supervised learning algorithms. Our key findings are that a pre-trained encoder
substantially improves 1) both accuracy under no attacks and certified security
guarantees against data poisoning and backdoor attacks of state-of-the-art
secure learning algorithms (i.e., bagging and KNN), 2) certified security
guarantees of randomized smoothing against adversarial examples without
sacrificing its accuracy under no attacks, 3) accuracy of differentially
private classifiers, and 4) accuracy and/or efficiency of exact machine
unlearning.
