  Modern machine learning (ML) ecosystems offer a surging number of ML
frameworks and code repositories that can greatly facilitate the development of
ML models. Today, even ordinary data holders who are not ML experts can apply
off-the-shelf codebase to build high-performance ML models on their data, many
of which are sensitive in nature (e.g., clinical records).
  In this work, we consider a malicious ML provider who supplies model-training
code to the data holders, does not have access to the training process, and has
only black-box query access to the resulting model. In this setting, we
demonstrate a new form of membership inference attack that is strictly more
powerful than prior art. Our attack empowers the adversary to reliably
de-identify all the training samples (average >99% attack<EMAIL_ADDRESS>FPR), and
the compromised models still maintain competitive performance as their
uncorrupted counterparts (average <1% accuracy drop). Moreover, we show that
the poisoned models can effectively disguise the amplified membership leakage
under common membership privacy auditing, which can only be revealed by a set
of secret samples known by the adversary.
  Overall, our study not only points to the worst-case membership privacy
leakage, but also unveils a common pitfall underlying existing privacy auditing
methods, which calls for future efforts to rethink the current practice of
auditing membership privacy in machine learning models.
