  In this work, we propose a method that combines two popular research areas by
injecting linguistic structures into pre-trained language models in the
parameter-efficient fine-tuning (PEFT) setting. In our approach, parallel
adapter modules encoding different linguistic structures are combined using a
novel Mixture-of-Linguistic-Experts architecture, where Gumbel-Softmax gates
are used to determine the importance of these modules at each layer of the
model. To reduce the number of parameters, we first train the model for a fixed
small number of steps before pruning the experts based on their importance
scores. Our experiment results with three different pre-trained models show
that our approach can outperform state-of-the-art PEFT methods with a
comparable number of parameters. In addition, we provide additional analysis to
examine the experts selected by each model at each layer to provide insights
for future studies.
