  Whether a word was bawled, whispered, or yelped, captions will typically
represent it in the same way. If they are your only way to access what is being
said, subjective nuances expressed in the voice will be lost. Since so much of
communication is carried by these nuances, we posit that if captions are to be
used as an accurate representation of speech, embedding visual representations
of paralinguistic qualities into captions could help readers use them to better
understand speech beyond its mere textual content. This paper presents a model
for processing vocal prosody (its loudness, pitch, and duration) and mapping it
into visual dimensions of typography (respectively, font-weight, baseline
shift, and letter-spacing), creating a visual representation of these lost
vocal subtleties that can be embedded directly into the typographical form of
text. An evaluation was carried out where participants were exposed to this
speech-modulated typography and asked to match it to its originating audio,
presented between similar alternatives. Participants (n=117) were able to
correctly identify the original audios with an average accuracy of 65%, with no
significant difference when showing them modulations as animated or static
text. Additionally, participants' comments showed their mental models of
speech-modulated typography varied widely.
