  Logs are important in modern software development with runtime information.
Log parsing is the first step in many log-based analyses, that involve
extracting structured information from unstructured log data. Traditional log
parsers face challenges in accurately parsing logs due to the diversity of log
formats, which directly impacts the performance of downstream log-analysis
tasks. In this paper, we explore the potential of using Large Language Models
(LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on
generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small,
Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16
open-source systems shows that LLMParser achieves statistically significantly
higher parsing accuracy than state-of-the-art parsers (a 96% average parsing
accuracy). We further conduct a comprehensive empirical analysis on the effect
of training size, model size, and pre-training LLM on log parsing accuracy. We
find that smaller LLMs may be more effective than more complex LLMs; for
instance where Flan-T5-base achieves comparable results as LLaMA-7B with a
shorter inference time. We also find that using LLMs pre-trained using logs
from other systems does not always improve parsing accuracy. While using
pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA
results in a decrease (decrease by almost 55% in group accuracy). In short, our
study provides empirical evidence for using LLMs for log parsing and highlights
the limitations and future research direction of LLM-based log parsers.
