  Privacy and security are major concerns when communicating speech signals to
cloud services such as automatic speech recognition (ASR) and speech emotion
recognition (SER). Existing solutions for speech anonymization mainly focus on
voice conversion or voice modification to convert a raw utterance into another
one with similar content but different, or no, identity-related information.
However, an alternative approach to share speech data under the form of
privacy-preserving representation has been largely under-explored. In this
paper, we propose a speech anonymization framework that achieves privacy via
noise perturbation to a selected subset of the high-utility representations
extracted using a pre-trained speech encoder. The subset is chosen with a
Transformer-based privacy-risk saliency estimator. We validate our framework on
four tasks, namely, Automatic Speaker Verification (ASV), ASR, SER and Intent
Classification (IC) for privacy and utility assessment. Experimental results
show that our approach is able to achieve a competitive, or even better,
utility compared to the speech anonymization baselines from the
VoicePrivacy2022 Challenges, providing the same level of privacy. Moreover, the
easily-controlled amount of perturbation allows our framework to have a
flexible range of privacy-utility trade-offs without re-training any component.
