  Transfer learning is a useful machine learning framework that allows one to
build task-specific models (student models) without significantly incurring
training costs using a single powerful model (teacher model) pre-trained with a
large amount of data. The teacher model may contain private data, or interact
with private inputs. We investigate if one can leak or infer such private
information without interacting with the teacher model directly. We describe
such inference attacks in the context of face recognition, an application of
transfer learning that is highly sensitive to personal privacy.
  Under black-box and realistic settings, we show that existing inference
techniques are ineffective, as interacting with individual training instances
through the student models does not reveal information about the teacher. We
then propose novel strategies to infer from aggregate-level information.
Consequently, membership inference attacks on the teacher model are shown to be
possible, even when the adversary has access only to the student models.
  We further demonstrate that sensitive attributes can be inferred, even in the
case where the adversary has limited auxiliary information. Finally, defensive
strategies are discussed and evaluated. Our extensive study indicates that
information leakage is a real privacy threat to the transfer learning framework
widely used in real-life situations.
