  Pre-trained language models (PLMs) are known to be overly parameterized and
have significant redundancy, indicating a small degree of freedom of the PLMs.
Motivated by the observation, in this paper, we study the problem of
re-parameterizing and fine-tuning PLMs from a new perspective: Discovery of
intrinsic task-specific subspace. Specifically, by exploiting the dynamics of
the fine-tuning process for a given task, the parameter optimization trajectory
is learned to uncover its intrinsic task-specific subspace. A key finding is
that PLMs can be effectively fine-tuned in the subspace with a small number of
free parameters. Beyond, we observe some outlier dimensions emerging during
fine-tuning in the subspace. Disabling these dimensions degrades the model
performance significantly. This suggests that these dimensions are crucial to
induce task-specific knowledge to downstream tasks.
