  General-purpose language models have demonstrated impressive capabilities,
performing on par with state-of-the-art approaches on a range of downstream
natural language processing (NLP) tasks and benchmarks when inferring
instructions from very few examples. Here, we evaluate the multilingual skills
of the GPT and T5 models in conducting multi-class classification on
non-English languages without any parameter updates. We show that, given a few
English examples as context, pre-trained language models can predict not only
English test samples but also non-English ones. Finally, we find the in-context
few-shot cross-lingual prediction results of language models are significantly
better than random prediction, and they are competitive compared to the
existing state-of-the-art cross-lingual models.
