  Neural Machine Translation (NMT) models usually use large target vocabulary
sizes to capture most of the words in the target language. The vocabulary size
is a big factor when decoding new sentences as the final softmax layer
normalizes over all possible target words. To address this problem, it is
widely common to restrict the target vocabulary with candidate lists based on
the source sentence. Usually, the candidate lists are a combination of external
word-to-word aligner, phrase table entries or most frequent words. In this
work, we propose a simple and yet novel approach to learn candidate lists
directly from the attention layer during NMT training. The candidate lists are
highly optimized for the current NMT model and do not need any external
computation of the candidate pool. We show significant decoding speedup
compared with using the entire vocabulary, without losing any translation
quality for two language pairs.
