  Speaker anonymization aims to conceal a speaker's identity while preserving
content information in speech. Current mainstream neural-network speaker
anonymization systems disentangle speech into prosody-related, content, and
speaker representations. The speaker representation is then anonymized by a
selection-based speaker anonymizer that uses a mean vector over a set of
randomly selected speaker vectors from an external pool of English speakers.
However, the resulting anonymized vectors are subject to severe privacy leakage
against powerful attackers, reduction in speaker diversity, and language
mismatch problems for unseen-language speaker anonymization. To generate
diverse, language-neutral speaker vectors, this paper proposes an anonymizer
based on an orthogonal Householder neural network (OHNN). Specifically, the
OHNN acts like a rotation to transform the original speaker vectors into
anonymized speaker vectors, which are constrained to follow the distribution
over the original speaker vector space. A basic classification loss is
introduced to ensure that anonymized speaker vectors from different speakers
have unique speaker identities. To further protect speaker identities, an
improved classification loss and similarity loss are used to push
original-anonymized sample pairs away from each other. Experiments on
VoicePrivacy Challenge datasets in English and the \textit{AISHELL-3} dataset
in Mandarin demonstrate the proposed anonymizer's effectiveness.
