  Cross-lingual knowledge transfer, especially between high- and low-resource
languages, remains a challenge in natural language processing (NLP). This study
offers insights for improving cross-lingual NLP applications through the
combination of parameter-efficient fine-tuning methods. We systematically
explore strategies for enhancing this cross-lingual transfer through the
incorporation of language-specific and task-specific adapters and soft prompts.
We present a detailed investigation of various combinations of these methods,
exploring their efficiency across six languages, focusing on three low-resource
languages, including the to our knowledge first use of soft language prompts.
Our findings demonstrate that in contrast to claims of previous work, a
combination of language and task adapters does not always work best; instead,
combining a soft language prompt with a task adapter outperforms other
configurations in many cases.
