  Recent interest has surged in employing Large Language Models (LLMs) for
machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023).
Most prior studies primarily focus on optimizing translation quality, with
limited attention to understanding the specific aspects of ICL that influence
the said quality. To this end, we perform the first of its kind, an exhaustive
study of in-context learning for machine translation. We first establish that
ICL is primarily example-driven and not instruction-driven. Following this, we
conduct an extensive exploration of various aspects of the examples to
understand their influence on downstream performance. Our analysis includes
factors such as quality and quantity of demonstrations, spatial proximity, and
source versus target originality. Further, we also investigate challenging
scenarios involving indirectness and misalignment of examples to understand the
limits of ICL. While we establish the significance of the quality of the target
distribution over the source distribution of demonstrations, we further observe
that perturbations sometimes act as regularizers, resulting in performance
improvements. Surprisingly, ICL does not necessitate examples from the same
task, and a related task with the same target distribution proves sufficient.
We hope that our study acts as a guiding resource for considerations in
utilizing ICL for MT. Our code is available on
https://github.com/PranjalChitale/in-context-mt-analysis.
