  Multilingual Neural Machine Translation has been showing great success using
transformer models. Deploying these models is challenging because they usually
require large vocabulary (vocab) sizes for various languages. This limits the
speed of predicting the output tokens in the last vocab projection layer. To
alleviate these challenges, this paper proposes a fast vocabulary projection
method via clustering which can be used for multilingual transformers on GPUs.
First, we offline split the vocab search space into disjoint clusters given the
hidden context vector of the decoder output, which results in much smaller
vocab columns for vocab projection. Second, at inference time, the proposed
method predicts the clusters and candidate active tokens for hidden context
vectors at the vocab projection. This paper also includes analysis of different
ways of building these clusters in multilingual settings. Our results show
end-to-end speed gains in float16 GPU inference up to 25% while maintaining the
BLEU score and slightly increasing memory cost. The proposed method speeds up
the vocab projection step itself by up to 2.6x. We also conduct an extensive
human evaluation to verify the proposed method preserves the quality of the
translations from the original model.
