  Neural models are known to be over-parameterized, and recent work has shown
that sparse text-to-speech (TTS) models can outperform dense models. Although a
plethora of sparse methods has been proposed for other domains, such methods
have rarely been applied in TTS. In this work, we seek to answer the question:
what are the characteristics of selected sparse techniques on the performance
and model complexity? We compare a Tacotron2 baseline and the results of
applying five techniques. We then evaluate the performance via the factors of
naturalness, intelligibility and prosody, while reporting model size and
training time. Complementary to prior research, we find that pruning before or
during training can achieve similar performance to pruning after training and
can be trained much faster, while removing entire neurons degrades performance
much more than removing parameters. To our best knowledge, this is the first
work that compares sparsity paradigms in text-to-speech synthesis.
