  The proliferation of misinformation and harmful narratives in online
discourse has underscored the critical need for effective Counter Narrative
(CN) generation techniques. However, existing automatic evaluation methods
often lack interpretability and fail to capture the nuanced relationship
between generated CNs and human perception. Aiming to achieve a higher
correlation with human judgments, this paper proposes a novel approach to asses
generated CNs that consists on the use of a Large Language Model (LLM) as a
evaluator. By comparing generated CNs pairwise in a tournament-style format, we
establish a model ranking pipeline that achieves a correlation of $0.88$ with
human preference. As an additional contribution, we leverage LLMs as zero-shot
(ZS) CN generators and conduct a comparative analysis of chat, instruct, and
base models, exploring their respective strengths and limitations. Through
meticulous evaluation, including fine-tuning experiments, we elucidate the
differences in performance and responsiveness to domain-specific data. We
conclude that chat-aligned models in ZS are the best option for carrying out
the task, provided they do not refuse to generate an answer due to security
concerns.
