  Given a document in a source language, cross-lingual summarization (CLS) aims
to generate a summary in a different target language. Recently, the emergence
of Large Language Models (LLMs), such as GPT-3.5, ChatGPT and GPT-4, has
attracted wide attention from the computational linguistics community. However,
it is not yet known the performance of LLMs on CLS. In this report, we
empirically use various prompts to guide LLMs to perform zero-shot CLS from
different paradigms (i.e., end-to-end and pipeline), and provide a preliminary
evaluation on the generated summaries. We find that ChatGPT and GPT-4
originally prefer to produce lengthy summaries with detailed information. These
two LLMs can further balance informativeness and conciseness with the help of
an interactive prompt, significantly improving their CLS performance.
Experimental results on three widely-used CLS datasets show that GPT-4 achieves
state-of-the-art zero-shot CLS performance, and performs competitively compared
with the fine-tuned mBART-50. Moreover, we also find some multi-lingual and
bilingual LLMs (i.e., BLOOMZ, ChatGLM-6B, Vicuna-13B and ChatYuan) have limited
zero-shot CLS ability. Due to the composite nature of CLS, which requires
models to perform summarization and translation simultaneously, accomplishing
this task in a zero-shot manner is even a challenge for LLMs. Therefore, we
sincerely hope and recommend future LLM research could use CLS as a testbed.
