  Large language models have demonstrated parallel and even superior
translation performance compared to neural machine translation (NMT) systems.
However, existing comparative studies between them mainly rely on automated
metrics, raising questions into the feasibility of these metrics and their
alignment with human judgment. The present study investigates the convergences
and divergences between automated metrics and human evaluation in assessing the
quality of machine translation from ChatGPT and three NMT systems. To perform
automatic assessment, four automated metrics are employed, while human
evaluation incorporates the DQF-MQM error typology and six rubrics. Notably,
automatic assessment and human evaluation converge in measuring formal fidelity
(e.g., error rates), but diverge when evaluating semantic and pragmatic
fidelity, with automated metrics failing to capture the improvement of
ChatGPT's translation brought by prompt engineering. These results underscore
the indispensable role of human judgment in evaluating the performance of
advanced translation tools at the current stage.
