  Large Language Models (LLMs) have shown promise in multiple software
engineering tasks including code generation, program repair, code
summarisation, and test generation. Fault localisation is instrumental in
enabling automated debugging and repair of programs and was prominently
featured as a highlight during the launch event of ChatGPT-4. Nevertheless, the
performance of LLMs compared to state-of-the-art methods, as well as the impact
of prompt design and context length on their efficacy, remains unclear. To fill
this gap, this paper presents an in-depth investigation into the capability of
ChatGPT-3.5 and ChatGPT-4, the two state-of-the-art LLMs, on fault
localisation. Using the widely-adopted large-scale Defects4J dataset, we
compare the two LLMs with the existing fault localisation techniques. We also
investigate the consistency of LLMs in fault localisation, as well as how
prompt engineering and the length of code context affect the fault localisation
effectiveness.
  Our findings demonstrate that within function-level context, ChatGPT-4
outperforms all the existing fault localisation methods. Additional error logs
can further improve ChatGPT models' localisation accuracy and consistency, with
an average 46.9% higher accuracy over the state-of-the-art baseline SmartFL on
the Defects4J dataset in terms of TOP-1 metric. However, when the code context
of the Defects4J dataset expands to the class-level, ChatGPT-4's performance
suffers a significant drop, with 49.9% lower accuracy than SmartFL under TOP-1
metric. These observations indicate that although ChatGPT can effectively
localise faults under specific conditions, limitations are evident. Further
research is needed to fully harness the potential of LLMs like ChatGPT for
practical fault localisation applications.
