  Evaluating Text Style Transfer (TST) is a complex task due to its
multifaceted nature. The quality of the generated text is measured based on
challenging factors, such as style transfer accuracy, content preservation, and
overall fluency. While human evaluation is considered to be the gold standard
in TST assessment, it is costly and often hard to reproduce. Therefore,
automated metrics are prevalent in these domains. Nevertheless, it remains
unclear whether these automated metrics correlate with human evaluations.
Recent strides in Large Language Models (LLMs) have showcased their capacity to
match and even exceed average human performance across diverse, unseen tasks.
This suggests that LLMs could be a feasible alternative to human evaluation and
other automated metrics in TST evaluation. We compare the results of different
LLMs in TST using multiple input prompts. Our findings highlight a strong
correlation between (even zero-shot) prompting and human evaluation, showing
that LLMs often outperform traditional automated metrics. Furthermore, we
introduce the concept of prompt ensembling, demonstrating its ability to
enhance the robustness of TST evaluation. This research contributes to the
ongoing evaluation of LLMs in diverse tasks, offering insights into successful
outcomes and areas of limitation.
