  Speaker adaptation systems face privacy concerns, for such systems are
trained on private datasets and often overfitting. This paper demonstrates that
an attacker can extract speaker information by querying speaker-adapted speech
recognition (ASR) systems. We focus on the speaker information of a
transformer-based ASR and propose GhostVec, a simple and efficient attack
method to extract the speaker information from an encoder-decoder-based ASR
system without any external speaker verification system or natural human voice
as a reference. To make our results quantitative, we pre-process GhostVec using
singular value decomposition (SVD) and synthesize it into waveform. Experiment
results show that the synthesized audio of GhostVec reaches 10.83\% EER and
0.47 minDCF with target speakers, which suggests the effectiveness of the
proposed method. We hope the preliminary discovery in this study to catalyze
future speech recognition research on privacy-preserving topics.
