  Neural sequence-to-sequence automatic speech recognition (ASR) systems are in
principle open vocabulary systems, when using appropriate modeling units. In
practice, however, they often fail to recognize words not seen during training,
e.g., named entities, numbers or technical terms. To alleviate this problem,
Huber et al. proposed to supplement an end-to-end ASR system with a word/phrase
memory and a mechanism to access this memory to recognize the words and phrases
correctly. In this paper we study, a) methods to acquire important words for
this memory dynamically and, b) the trade-off between improvement in
recognition accuracy of new words and the potential danger of false alarms for
those added words. We demonstrate significant improvements in the detection
rate of new words with only a minor increase in false alarms (F1 score 0.30
$\rightarrow$ 0.80), when using an appropriate number of new words. In
addition, we show that important keywords can be extracted from supporting
documents and used effectively.
