  This paper emphasizes the importance of reporting experiment details in
subjective evaluations and demonstrates how such details can significantly
impact evaluation results in the field of speech synthesis. Through an analysis
of 80 papers presented at INTERSPEECH 2022, we find a lack of thorough
reporting on critical details such as evaluator recruitment and filtering,
instructions and payments, and the geographic and linguistic backgrounds of
evaluators. To illustrate the effect of these details on evaluation outcomes,
we conducted mean opinion score (MOS) tests on three well-known TTS systems
under different evaluation settings and we obtain at least three distinct
rankings of TTS models. We urge the community to report experiment details in
subjective evaluations to improve the reliability and interpretability of
experimental results.
