  Even though large language models (LLMs) have demonstrated remarkable
capability in solving various natural language tasks, the capability of an LLM
to follow human instructions is still a concern. Recent works have shown great
improvements in the instruction-following capability via additional training
for instruction-following tasks. However, the mechanisms responsible for
effective instruction-following capabilities remain inadequately understood.
Here, we introduce a simplified instruction-following task and use synthetic
datasets to analyze a Transformer-based causal language model. Our findings
suggest that the model learns task-specific information by clustering data
within its hidden space, with this clustering process evolving dynamically
during learning. We also demonstrate how this phenomenon assists the model in
handling unseen instances, and validate our results in a more realistic
setting. Furthermore, we present inspired applications regarding pre-training
and alignment.
