  Privacy-preserving voice protection approaches primarily suppress
privacy-related information derived from paralinguistic attributes while
preserving the linguistic content. Existing solutions focus on single-speaker
scenarios. However, they lack practicality for real-world applications, i.e.,
multi-speaker scenarios. In this paper, we present an initial attempt to
provide a multi-speaker anonymization benchmark by defining the task and
evaluation protocol, proposing benchmarking solutions, and discussing the
privacy leakage of overlapping conversations. Specifically, ideal multi-speaker
anonymization should preserve the number of speakers and the turn-taking
structure of the conversation, ensuring accurate context conveyance while
maintaining privacy. To achieve that, a cascaded system uses speaker
diarization to aggregate the speech of each speaker and speaker anonymization
to conceal speaker privacy and preserve speech content. Additionally, we
propose two conversation-level speaker vector anonymization methods to improve
the utility further. Both methods aim to make the original and corresponding
pseudo-speaker identities of each speaker unlinkable while preserving or even
improving the distinguishability among pseudo-speakers in a conversation. The
first method minimizes the differential similarity across speaker pairs in the
original and anonymized conversations to maintain original speaker
relationships in the anonymized version. The other method minimizes the
aggregated similarity across anonymized speakers to achieve better
differentiation between speakers. Experiments conducted on both non-overlap
simulated and real-world datasets demonstrate the effectiveness of the
multi-speaker anonymization system with the proposed speaker anonymizers.
Additionally, we analyzed overlapping speech regarding privacy leakage and
provide potential solutions.
