  A recent family of techniques, dubbed lightweight fine-tuning methods,
facilitates parameter-efficient transfer learning by updating only a small set
of additional parameters while keeping the parameters of the pretrained
language model frozen. While proven to be an effective method, there are no
existing studies on if and how such knowledge of the downstream fine-tuning
approach should affect the pretraining stage. In this work, we show that taking
the ultimate choice of fine-tuning method into consideration boosts the
performance of parameter-efficient fine-tuning. By relying on
optimization-based meta-learning using MAML with certain modifications for our
distinct purpose, we prime the pretrained model specifically for
parameter-efficient fine-tuning, resulting in gains of up to 1.7 points on
cross-lingual NER fine-tuning. Our ablation settings and analyses further
reveal that the tweaks we introduce in MAML are crucial for the attained gains.
