  Adapting pre-trained foundation models for various downstream tasks has been
prevalent in artificial intelligence. Due to the vast number of tasks and high
costs, adjusting all parameters becomes unfeasible. To mitigate this, several
fine-tuning techniques have been developed to update the pre-trained model
weights in a more resource-efficient manner, such as through low-rank
adjustments. Yet, almost all of these methods focus on linear weights,
neglecting the intricacies of parameter spaces in higher dimensions like 4D.
Alternatively, some methods can be adapted for high-dimensional parameter space
by compressing changes in the original space into two dimensions and then
employing low-rank matrix decomposition. However, these approaches destructs
the structural integrity of the involved high-dimensional spaces. To tackle the
diversity of dimensional spaces across different foundation models and provide
a more precise representation of the changes within these spaces, this paper
introduces a generalized parameter-efficient fine-tuning framework, FLoRA,
designed for various dimensional parameter space. Specifically, utilizing
Tucker decomposition, FLoRA asserts that changes in each dimensional parameter
space are based on a low-rank core space which maintains the consistent
topological structure with the original space. It then models the changes
through this core space alongside corresponding weights to reconstruct
alterations in the original space. FLoRA effectively preserves the structural
integrity of the change of original N-dimensional parameter space, meanwhile
decomposes it via low-rank tensor decomposition. Extensive experiments on
computer vision, natural language processing and multi-modal tasks validate
FLoRA's effectiveness. Codes are available at
https://github.com/SJTU-DeepVisionLab/FLoRA.
