  In a model inversion (MI) attack, an adversary abuses access to a machine
learning (ML) model to infer and reconstruct private training data. Remarkable
progress has been made in the white-box and black-box setups, where the
adversary has access to the complete model or the model's soft output
respectively. However, there is very limited study in the most challenging but
practically important setup: Label-only MI attacks, where the adversary only
has access to the model's predicted label (hard label) without confidence
scores nor any other model information.
  In this work, we propose LOKT, a novel approach for label-only MI attacks.
Our idea is based on transfer of knowledge from the opaque target model to
surrogate models. Subsequently, using these surrogate models, our approach can
harness advanced white-box attacks. We propose knowledge transfer based on
generative modelling, and introduce a new model, Target model-assisted ACGAN
(T-ACGAN), for effective knowledge transfer. Our method casts the challenging
label-only MI into the more tractable white-box setup. We provide analysis to
support that surrogate models based on our approach serve as effective proxies
for the target model for MI. Our experiments show that our method significantly
outperforms existing SOTA Label-only MI attack by more than 15% across all MI
benchmarks. Furthermore, our method compares favorably in terms of query
budget. Our study highlights rising privacy threats for ML models even when
minimal information (i.e., hard labels) is exposed. Our study highlights rising
privacy threats for ML models even when minimal information (i.e., hard labels)
is exposed. Our code, demo, models and reconstructed data are available at our
project page: https://ngoc-nguyen-0.github.io/lokt/
