  While following different technical routes, both low-rank and orthogonal
adaptation techniques can efficiently adapt large-scale pre-training models in
specific tasks or domains based on a small piece of trainable parameters. In
this study, we bridge the gap between these two techniques, proposing a simple
but effective adaptation method based on Householder reflections. Given a
pre-trained model, our method fine-tunes its layers by multiplying each frozen
weight matrix with an orthogonal matrix constructed by a chain of learnable
Householder reflections (HRs). This HR-based orthogonal fine-tuning is
equivalent to an adaptive low-rank adaptation. Moreover, we show that the
orthogonality of the reflection planes corresponding to the HRs impacts the
model capacity and regularity. The analysis motivates us to regularize the
orthogonality of the HRs, leading to different implementations of the proposed
Householder reflection adaptation (HRA) method. Compared with state-of-the-art
methods, HRA achieves superior performance with fewer learnable parameters when
adapting large language models and conditional image generators. The code is
available at https://github.com/DaShenZi721/HRA
