  Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a
source language to make predictions in another language, often with a
performance loss. To alleviate this, additional improvements can be achieved
through subsequent adaptation using examples in the target language. In this
paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer
in the classification task by introducing In-Context Cross-lingual Transfer
(IC-XLT). The novel concept involves training a model to learn from context
examples and subsequently adapting it during inference to a target language by
prepending a One-Shot context demonstration in that language. Our results show
that IC-XLT successfully leverages target-language examples to improve the
cross-lingual capabilities of the evaluated mT5 model, outperforming
prompt-based models in the Zero and Few-shot scenarios adapted through
fine-tuning. Moreover, we show that when source-language data is limited, the
fine-tuning framework employed for IC-XLT performs comparably to prompt-based
fine-tuning with significantly more training data in the source language.
