  Among the many tasks that Large Language Models (LLMs) have revolutionized is
text classification. Current text classification paradigms, however, rely
solely on the output of the final layer in the LLM, with the rich information
contained in internal neurons largely untapped. In this study, we present SPIN:
a model-agnostic framework that sparsifies and integrates internal neurons of
intermediate layers of LLMs for text classification. Specifically, SPIN
sparsifies internal neurons by linear probing-based salient neuron selection
layer by layer, avoiding noise from unrelated neurons and ensuring efficiency.
The cross-layer salient neurons are then integrated to serve as multi-layered
features for the classification head. Extensive experimental results show our
proposed SPIN significantly improves text classification accuracy, efficiency,
and interpretability.
