  Massive Over-activation Yielded Uplifts(MOYU) is an inherent property of
large language models, and dynamic activation(DA) based on the MOYU property is
a clever yet under-explored strategy designed to accelerate inference in these
models. Existing methods that utilize MOYU often face a significant 'Impossible
Trinity': struggling to simultaneously maintain model performance, enhance
inference speed, and extend applicability across various architectures. Due to
the theoretical ambiguities surrounding MOYU, this paper elucidates the root
cause of the MOYU property and outlines the mechanisms behind two primary
limitations encountered by current DA methods: 1) history-related activation
uncertainty, and 2) semantic-irrelevant activation inertia. Our analysis not
only underscores the limitations of current dynamic activation strategies
within large-scale LLaMA models but also proposes opportunities for refining
the design of future sparsity schemes.
