  We investigate the potential of ChatGPT as a multidimensional evaluator for
the task of \emph{Text Style Transfer}, alongside, and in comparison to,
existing automatic metrics as well as human judgements. We focus on a zero-shot
setting, i.e. prompting ChatGPT with specific task instructions, and test its
performance on three commonly-used dimensions of text style transfer
evaluation: style strength, content preservation, and fluency. We perform a
comprehensive correlation analysis for two transfer directions (and overall) at
different levels. Compared to existing automatic metrics, ChatGPT achieves
competitive correlations with human judgments. These preliminary results are
expected to provide a first glimpse into the role of large language models in
the multidimensional evaluation of stylized text generation.
