  Instruction Tuning (IT), the process of training large language models (LLMs)
using instruction-response pairs, has emerged as the predominant method for
transforming base pre-trained LLMs into open-domain conversational agents.
While IT has achieved notable success and widespread adoption, its limitations
and shortcomings remain underexplored. In this paper, through rigorous
experiments and an in-depth analysis of the changes LLMs undergo through IT, we
reveal various limitations of IT. In particular, we show that (1) IT fails to
enhance knowledge or skills in LLMs. LoRA fine-tuning is limited to learning
response initiation and style tokens, and full-parameter fine-tuning leads to
knowledge degradation. (2) Copying response patterns from IT datasets derived
from knowledgeable sources leads to a decline in response quality. (3)
Full-parameter fine-tuning increases hallucination by inaccurately borrowing
tokens from conceptually similar instances in the IT dataset for generating
responses. (4) Popular methods to improve IT do not lead to performance
improvements over a simple LoRA fine-tuned model. Our findings reveal that
responses generated solely from pre-trained knowledge consistently outperform
responses by models that learn any form of new knowledge from IT on open-source
datasets. We hope the insights and challenges revealed in this paper inspire
future work in related directions.
