  Speech synthesis has recently seen significant improvements in fidelity,
driven by the advent of neural vocoders and neural prosody generators. However,
these systems lack intuitive user controls over prosody, making them unable to
rectify prosody errors (e.g., misplaced emphases and contextually inappropriate
emotions) or generate prosodies with diverse speaker excitement levels and
emotions. We address these limitations with a user-controllable, context-aware
neural prosody generator. Given a real or synthesized speech recording, our
model allows a user to input prosody constraints for certain time frames and
generates the remaining time frames from input text and contextual prosody. We
also propose a pitch-shifting neural vocoder to modify input speech to match
the synthesized prosody. Through objective and subjective evaluations we show
that we can successfully incorporate user control into our prosody generation
model without sacrificing the overall naturalness of the synthesized speech.
