  Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task
has been a well-known paradigm in Natural Language Processing. However, with
the ever-growing size of PLMs, training the entire model on several downstream
tasks becomes very expensive and resource-hungry. Recently, different Parameter
Efficient Tuning (PET) techniques are proposed to improve the efficiency of
fine-tuning PLMs. One popular category of PET methods is the low-rank
adaptation methods which insert learnable truncated SVD modules into the
original model either sequentially or in parallel. However, low-rank
decomposition suffers from limited representation power. In this work, we
address this problem using the Kronecker product instead of the low-rank
representation. We introduce KronA, a Kronecker product-based adapter module
for efficient fine-tuning of Transformer-based PLMs. We apply the proposed
methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the
Kronecker-based modules can outperform state-of-the-art PET methods.
