  We present an approach for unsupervised learning of speech representation
disentangling contents and styles. Our model consists of: (1) a local encoder
that captures per-frame information; (2) a global encoder that captures
per-utterance information; and (3) a conditional decoder that reconstructs
speech given local and global latent variables. Our experiments show that (1)
the local latent variables encode speech contents, as reconstructed speech can
be recognized by ASR with low word error rates (WER), even with a different
global encoding; (2) the global latent variables encode speaker style, as
reconstructed speech shares speaker identity with the source utterance of the
global encoding. Additionally, we demonstrate an useful application from our
pre-trained model, where we can train a speaker recognition model from the
global latent variables and achieve high accuracy by fine-tuning with as few
data as one label per speaker.
