  For articulatory-to-acoustic mapping, typically only limited parallel
training data is available, making it impossible to apply fully end-to-end
solutions like Tacotron2. In this paper, we experimented with transfer learning
and adaptation of a Tacotron2 text-to-speech model to improve the final
synthesis quality of ultrasound-based articulatory-to-acoustic mapping with a
limited database. We use a multi-speaker pre-trained Tacotron2 TTS model and a
pre-trained WaveGlow neural vocoder. The articulatory-to-acoustic conversion
contains three steps: 1) from a sequence of ultrasound tongue image recordings,
a 3D convolutional neural network predicts the inputs of the pre-trained
Tacotron2 model, 2) the Tacotron2 model converts this intermediate
representation to an 80-dimensional mel-spectrogram, and 3) the WaveGlow model
is applied for final inference. This generated speech contains the timing of
the original articulatory data from the ultrasound recording, but the F0
contour and the spectral information is predicted by the Tacotron2 model. The
F0 values are independent of the original ultrasound images, but represent the
target speaker, as they are inferred from the pre-trained Tacotron2 model. In
our experiments, we demonstrated that the synthesized speech quality is more
natural with the proposed solutions than with our earlier model.
