  Machine-generated speech is characterized by its limited or unnatural
emotional variation. Current text to speech systems generates speech with
either a flat emotion, emotion selected from a predefined set, average
variation learned from prosody sequences in training data or transferred from a
source style. We propose a text to speech(TTS) system, where a user can choose
the emotion of generated speech from a continuous and meaningful emotion space
(Arousal-Valence space). The proposed TTS system can generate speech from the
text in any speaker's style, with fine control of emotion. We show that the
system works on emotion unseen during training and can scale to previously
unseen speakers given his/her speech sample. Our work expands the horizon of
the state-of-the-art FastSpeech2 backbone to a multi-speaker setting and gives
it much-coveted continuous (and interpretable) affective control, without any
observable degradation in the quality of the synthesized speech.
