  Neural evaluation metrics derived for numerous speech generation tasks have
recently attracted great attention. In this paper, we propose SVSNet, the first
end-to-end neural network model to assess the speaker voice similarity between
converted speech and natural speech for voice conversion tasks. Unlike most
neural evaluation metrics that use hand-crafted features, SVSNet directly takes
the raw waveform as input to more completely utilize speech information for
prediction. SVSNet consists of encoder, co-attention, distance calculation, and
prediction modules and is trained in an end-to-end manner. The experimental
results on the Voice Conversion Challenge 2018 and 2020 (VCC2018 and VCC2020)
datasets show that SVSNet outperforms well-known baseline systems in the
assessment of speaker similarity at the utterance and system levels.
