  Which phonemes convey more speaker traits is a long-standing question, and
various perception experiments were conducted with human subjects. For speaker
recognition, studies were conducted with the conventional statistical models
and the drawn conclusions are more or less consistent with the perception
results. However, which phonemes are more important with modern deep neural
models is still unexplored, due to the opaqueness of the decision process. This
paper conducts a novel study for the attribution of phonemes with two types of
deep speaker models that are based on TDNN and CNN respectively, from the
perspective of model explanation. Specifically, we conducted the study by two
post-explanation methods: LayerCAM and Time Align Occlusion (TAO). Experimental
results showed that: (1) At the population level, vowels are more important
than consonants, confirming the human perception studies. However, fricatives
are among the most unimportant phonemes, which contrasts with previous studies.
(2) At the speaker level, a large between-speaker variation is observed
regarding phoneme importance, indicating that whether a phoneme is important or
not is largely speaker-dependent.
