  Transformer models can face practical limitations due to their high
computational requirements. At the same time, such models exhibit significant
activation sparsity, which can be leveraged to reduce the inference cost by
converting parts of the network into equivalent Mixture-of-Experts (MoE)
layers. Despite the crucial role played by activation sparsity, its impact on
this process remains unexplored. In particular, we show that the efficiency of
the conversion can be significantly enhanced by a proper regularization of the
activation sparsity of the base model. Moreover, motivated by the high variance
of the number of activated neurons for different inputs, we introduce a more
effective dynamic-k expert selection rule that adjusts the number of executed
experts on a per-token basis. Finally, we extend this approach to multi-head
attention projections, which results in additional savings compared to only
converting the FFN blocks. The proposed method, Dense to Dynamic-$k$
Mixture-of-Experts (D2DMoE), outperforms existing approaches on common NLP and
vision tasks, allowing us to save up to 60% of inference cost without
significantly affecting model performance.
