  Speaker adaptation methods aim to create fair quality synthesis speech voice
font for target speakers while only limited resources available. Recently, as
deep neural networks based statistical parametric speech synthesis (SPSS)
methods become dominant in SPSS TTS back-end modeling, speaker adaptation under
the neural network based SPSS framework has also became an important task. In
this paper, linear networks (LN) is inserted in multiple neural network layers
and fine-tuned together with output layer for best speaker adaptation
performance. When adaptation data is extremely small, the low-rank plus
diagonal(LRPD) decomposition for LN is employed to make the adapted voice more
stable. Speaker adaptation experiments are conducted under a range of
adaptation utterances numbers. Moreover, speaker adaptation from 1) female to
female, 2) male to female and 3) female to male are investigated. Objective
measurement and subjective tests show that LN with LRPD decomposition performs
most stable when adaptation data is extremely limited, and our best speaker
adaptation (SA) model with only 200 adaptation utterances achieves comparable
quality with speaker dependent (SD) model trained with 1000 utterances, in both
naturalness and similarity to target speaker.
