  We propose Quantum-informed Tensor Adaptation (QuanTA), a novel,
easy-to-implement, fine-tuning method with no inference overhead for
large-scale pre-trained language models. By leveraging quantum-inspired methods
derived from quantum circuit structures, QuanTA enables efficient high-rank
fine-tuning, surpassing the limitations of Low-Rank Adaptation (LoRA)--low-rank
approximation may fail for complicated downstream tasks. Our approach is
theoretically supported by the universality theorem and the rank representation
theorem to achieve efficient high-rank adaptations. Experiments demonstrate
that QuanTA significantly enhances commonsense reasoning, arithmetic reasoning,
and scalability compared to traditional methods. Furthermore, QuanTA shows
superior performance with fewer trainable parameters compared to other
approaches and can be designed to integrate with existing fine-tuning
algorithms for further improvement, providing a scalable and efficient solution
for fine-tuning large language models and advancing state-of-the-art in natural
language processing.
