  Large language models (LLMs) that have been trained on multilingual but not
parallel text exhibit a remarkable ability to translate between languages. We
probe this ability in an in-depth study of the pathways language model (PaLM),
which has demonstrated the strongest machine translation (MT) performance among
similarly-trained LLMs to date. We investigate various strategies for choosing
translation examples for few-shot prompting, concluding that example quality is
the most important factor. Using optimized prompts, we revisit previous
assessments of PaLM's MT capabilities with more recent test sets, modern MT
metrics, and human evaluation, and find that its performance, while impressive,
still lags that of state-of-the-art supervised systems. We conclude by
providing an analysis of PaLM's MT output which reveals some interesting
properties and prospects for future work.
