  We propose an unsupervised speaker adaptation method inspired by the neural
Turing machine for end-to-end (E2E) automatic speech recognition (ASR). The
proposed model contains a memory block that holds speaker i-vectors extracted
from the training data and reads relevant i-vectors from the memory through an
attention mechanism. The resulting memory vector (M-vector) is concatenated to
the acoustic features or to the hidden layer activations of an E2E neural
network model. The E2E ASR system is based on the joint connectionist temporal
classification and attention-based encoder-decoder architecture. M-vector and
i-vector results are compared for inserting them at different layers of the
encoder neural network using the WSJ and TED-LIUM2 ASR benchmarks. We show that
M-vectors, which do not require an auxiliary speaker embedding extraction
system at test time, achieve similar word error rates (WERs) compared to
i-vectors for single speaker utterances and significantly lower WERs for
utterances in which there are speaker changes.
