  Log data have facilitated various tasks of software development and
maintenance, such as testing, debugging and diagnosing. Due to the unstructured
nature of logs, log parsing is typically required to transform log messages
into structured data for automated log analysis. Given the abundance of log
parsers that employ various techniques, evaluating these tools to comprehend
their characteristics and performance becomes imperative. Loghub serves as a
commonly used dataset for benchmarking log parsers, but it suffers from limited
scale and representativeness, posing significant challenges for studies to
comprehensively evaluate existing log parsers or develop new methods. This
limitation is particularly pronounced when assessing these log parsers for
production use. To address these limitations, we provide a new collection of
annotated log datasets, denoted Loghub-2.0, which can better reflect the
characteristics of log data in real-world software systems. Loghub-2.0
comprises 14 datasets with an average of 3.6 million log lines in each dataset.
Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art
log parsers in a more rigorous and practical setting. Particularly, we
introduce a new evaluation metric to mitigate the sensitivity of existing
metrics to imbalanced data distributions. We are also the first to investigate
the granular performance of log parsers on logs that represent rare system
events, offering in-depth details for software diagnosis. Accurately parsing
such logs is essential, yet it remains a challenge. We believe this work could
shed light on the evaluation and design of log parsers in practical settings,
thereby facilitating their deployment in production systems.
