  There is increasing interest in the adoption of LLMs in HCI research.
However, LLMs may often be regarded as a panacea because of their powerful
capabilities with an accompanying oversight on whether they are suitable for
their intended tasks. We contend that LLMs should be adopted in a critical
manner following rigorous evaluation. Accordingly, we present the evaluation of
an LLM in identifying logical fallacies that will form part of a digital
misinformation intervention. By comparing to a labeled dataset, we found that
GPT-4 achieves an accuracy of 0.79, and for our intended use case that excludes
invalid or unidentified instances, an accuracy of 0.90. This gives us the
confidence to proceed with the application of the LLM while keeping in mind the
areas where it still falls short. The paper describes our evaluation approach,
results and reflections on the use of the LLM for our intended task.
