  The conversion from text to speech relies on the accurate mapping from
linguistic to acoustic symbol sequences, for which current practice employs
recurrent statistical models like recurrent neural networks. Despite the good
performance of such models (in terms of low distortion in the generated
speech), their recursive structure tends to make them slow to train and to
sample from. In this work, we try to overcome the limitations of recursive
structure by using a module based on the transformer decoder network, designed
without recurrent connections but emulating them with attention and positioning
codes. Our results show that the proposed decoder network is competitive in
terms of distortion when compared to a recurrent baseline, whilst being
significantly faster in terms of CPU inference time. On average, it increases
Mel cepstral distortion between 0.1 and 0.3 dB, but it is over an order of
magnitude faster on average. Fast inference is important for the deployment of
speech synthesis systems on devices with restricted resources, like mobile
phones or embedded systems, where speaking virtual assistants are gaining
importance.
