  Recognizing whispered speech and converting it to normal speech creates many
possibilities for speech interaction. Because the sound pressure of whispered
speech is significantly lower than that of normal speech, it can be used as a
semi-silent speech interaction in public places without being audible to
others. Converting whispers to normal speech also improves the speech quality
for people with speech or hearing impairments. However, conventional speech
conversion techniques do not provide sufficient conversion quality or require
speaker-dependent datasets consisting of pairs of whispered and normal speech
utterances. To address these problems, we propose WESPER, a zero-shot,
real-time whisper-to-normal speech conversion mechanism based on
self-supervised learning. WESPER consists of a speech-to-unit (STU) encoder,
which generates hidden speech units common to both whispered and normal speech,
and a unit-to-speech (UTS) decoder, which reconstructs speech from the encoded
speech units. Unlike the existing methods, this conversion is user-independent
and does not require a paired dataset for whispered and normal speech. The UTS
decoder can reconstruct speech in any target speaker's voice from speech units,
and it requires only an unlabeled target speaker's speech data. We confirmed
that the quality of the speech converted from a whisper was improved while
preserving its natural prosody. Additionally, we confirmed the effectiveness of
the proposed approach to perform speech reconstruction for people with speech
or hearing disabilities. (project page: http://lab.rekimoto.org/projects/wesper
)
