  In recent years, text-to-speech (TTS) technology has witnessed impressive
advancements, particularly with large-scale training datasets, showcasing
human-level speech quality and impressive zero-shot capabilities on unseen
speakers. However, despite human subjective evaluations, such as the mean
opinion score (MOS), remaining the gold standard for assessing the quality of
synthetic speech, even state-of-the-art TTS approaches have kept human feedback
isolated from training that resulted in mismatched training objectives and
evaluation metrics. In this work, we investigate a novel topic of integrating
subjective human evaluation into the TTS training loop. Inspired by the recent
success of reinforcement learning from human feedback, we propose a
comprehensive sampling-annotating-learning framework tailored to TTS
optimization, namely uncertainty-aware optimization (UNO). Specifically, UNO
eliminates the need for a reward model or preference data by directly
maximizing the utility of speech generations while considering the uncertainty
that lies in the inherent variability in subjective human speech perception and
evaluations. Experimental results of both subjective and objective evaluations
demonstrate that UNO considerably improves the zero-shot performance of TTS
models in terms of MOS, word error rate, and speaker similarity. Additionally,
we present a remarkable ability of UNO that it can adapt to the desired
speaking style in emotional TTS seamlessly and flexibly.
