  Sparse activation, which selectively activates only an input-dependent set of
neurons in inference, is a useful technique to reduce the computing cost of
Large Language Models (LLMs) without retraining or adaptation efforts. However,
whether it can be applied to the recently emerging Small Language Models (SLMs)
remains questionable, because SLMs are generally less over-parameterized than
LLMs. In this paper, we aim to achieve sparse activation in SLMs. We first show
that the existing sparse activation schemes in LLMs that build on neurons'
output magnitudes cannot be applied to SLMs, and activating neurons based on
their attribution scores is a better alternative. Further, we demonstrated and
quantified the large errors of existing attribution metrics when being used for
sparse activation, due to the interdependency among attribution scores of
neurons across different layers. Based on these observations, we proposed a new
attribution metric that can provably correct such errors and achieve precise
sparse activation. Experiments over multiple popular SLMs and datasets show
that our approach can achieve 80% sparsification ratio with <5% model accuracy
loss, comparable to the sparse activation achieved in LLMs. The source code is
available at: https://github.com/pittisl/Sparse-Activation.
