  Machine learning (ML) has been widely adopted in various privacy-critical
applications, e.g., face recognition and medical image analysis. However,
recent research has shown that ML models are vulnerable to attacks against
their training data. Membership inference is one major attack in this domain:
Given a data sample and model, an adversary aims to determine whether the
sample is part of the model's training set. Existing membership inference
attacks leverage the confidence scores returned by the model as their inputs
(score-based attacks). However, these attacks can be easily mitigated if the
model only exposes the predicted label, i.e., the final model decision.
  In this paper, we propose decision-based membership inference attacks and
demonstrate that label-only exposures are also vulnerable to membership
leakage. In particular, we develop two types of decision-based attacks, namely
transfer attack, and boundary attack. Empirical evaluation shows that our
decision-based attacks can achieve remarkable performance, and even outperform
the previous score-based attacks in some cases. We further present new insights
on the success of membership inference based on quantitative and qualitative
analysis, i.e., member samples of a model are more distant to the model's
decision boundary than non-member samples. Finally, we evaluate multiple
defense mechanisms against our decision-based attacks and show that our two
types of attacks can bypass most of these defenses.
