  While large language models (LLMs) have demonstrated superior multi-task
capabilities, understanding the learning mechanisms behind this is still a
challenging problem. In this paper, we attempt to understand such mechanisms
from the perspective of neurons. Specifically, we detect task-sensitive neurons
in LLMs via gradient attribution on task-specific data. Through extensive
deactivation and fine-tuning experiments, we demonstrate that the detected
neurons are highly correlated with the given task, which we term as
task-specific neurons. With these identified task-specific neurons, we delve
into two common problems in multi-task learning and continuous learning:
Generalization and Catastrophic Forgetting. We find that the overlap of
task-specific neurons is strongly associated with generalization and
specialization across tasks. Interestingly, at certain layers of LLMs, there is
a high similarity in the parameters of different task-specific neurons, and
such similarity is highly correlated with the generalization performance.
Inspired by these findings, we propose a neuron-level continuous fine-tuning
method that only fine-tunes the current task-specific neurons during continuous
learning, and extensive experiments demonstrate the effectiveness of the
proposed method. Our study provides insights into the interpretability of LLMs
in multi-task learning.
