  With the advent of the era of foundation models, pre-training and fine-tuning
have become common paradigms. Recently, parameter-efficient fine-tuning has
garnered widespread attention due to its better balance between the number of
learnable parameters and performance. However, some current parameter-efficient
fine-tuning methods only model a single modality and lack the utilization of
structural knowledge in downstream tasks. To address this issue, this paper
proposes a multi-modal parameter-efficient fine-tuning method based on graph
networks. Each image is fed into a multi-modal large language model (MLLM) to
generate a text description. The image and its corresponding text description
are then processed by a frozen image encoder and text encoder to generate image
features and text features, respectively. A graph is constructed based on the
similarity of the multi-modal feature nodes, and knowledge and relationships
relevant to these features are extracted from each node. Additionally, Elastic
Weight Consolidation (EWC) regularization is incorporated into the loss
function to mitigate the problem of forgetting during task learning. The
proposed model achieves test accuracies on the OxfordPets, Flowers102, and
Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The
code is available at https://github.com/yunche0/GA-Net/tree/master.
