  While deep learning has made impressive progress in speech synthesis and
voice conversion, the assessment of the synthesized speech is still carried out
by human participants. Several recent papers have proposed deep-learning-based
assessment models and shown the potential to automate the speech quality
assessment. To improve the previously proposed assessment model, MOSNet, we
propose three models using cluster-based modeling methods: using a global
quality token (GQT) layer, using an Encoding Layer, and using both of them. We
perform experiments using the evaluation results of the Voice Conversion
Challenge 2018 to predict the mean opinion score of synthesized speech and
similarity score between synthesized speech and reference speech. The results
show that the GQT layer helps to predict human assessment better by
automatically learning the useful quality tokens for the task and that the
Encoding Layer helps to utilize frame-level scores more precisely.
