  Variational auto-encoder (VAE) is an effective neural network architecture to
disentangle a speech utterance into speaker identity and linguistic content
latent embeddings, then generate an utterance for a target speaker from that of
a source speaker. This is possible by concatenating the identity embedding of
the target speaker and the content embedding of the source speaker uttering a
desired sentence. In this work, we propose to improve VAE models with
self-attention and structural regularization (RGSM). Specifically, we found a
suitable location of VAE's decoder to add a self-attention layer for
incorporating non-local information in generating a converted utterance and
hiding the source speaker's identity. We applied relaxed group-wise splitting
method (RGSM) to regularize network weights and remarkably enhance
generalization performance.
  In experiments of zero-shot many-to-many voice conversion task on VCTK data
set, with the self-attention layer and relaxed group-wise splitting method, our
model achieves a gain of speaker classification accuracy on unseen speakers by
28.3\% while slightly improved conversion voice quality in terms of MOSNet
scores. Our encouraging findings point to future research on integrating more
variety of attention structures in VAE framework while controlling model size
and overfitting for advancing zero-shot many-to-many voice conversions.
