  Prompt-based tuning has been proven effective for pretrained language models
(PLMs). While most of the existing work focuses on the monolingual prompts, we
study the multilingual prompts for multilingual PLMs, especially in the
zero-shot cross-lingual setting. To alleviate the effort of designing different
prompts for multiple languages, we propose a novel model that uses a unified
prompt for all languages, called UniPrompt. Different from the discrete prompts
and soft prompts, the unified prompt is model-based and language-agnostic.
Specifically, the unified prompt is initialized by a multilingual PLM to
produce language-independent representation, after which is fused with the text
input. During inference, the prompts can be pre-computed so that no extra
computation cost is needed. To collocate with the unified prompt, we propose a
new initialization method for the target label word to further improve the
model's transferability across languages. Extensive experiments show that our
proposed methods can significantly outperform the strong baselines across
different languages. We release data and code to facilitate future research.
