  Logs are imperative in the development and maintenance process of many
software systems. They record detailed runtime information that allows
developers and support engineers to monitor their systems and dissect anomalous
behaviors and errors. The increasing scale and complexity of modern software
systems, however, make the volume of logs explodes. In many cases, the
traditional way of manual log inspection becomes impractical. Many recent
studies, as well as industrial tools, resort to powerful text search and
machine learning-based analytics solutions. Due to the unstructured nature of
logs, a first crucial step is to parse log messages into structured data for
subsequent analysis. In recent years, automated log parsing has been widely
studied in both academia and industry, producing a series of log parsers by
different techniques. To better understand the characteristics of these log
parsers, in this paper, we present a comprehensive evaluation study on
automated log parsing and further release the tools and benchmarks for easy
reuse. More specifically, we evaluate 13 log parsers on a total of 16 log
datasets spanning distributed systems, supercomputers, operating systems,
mobile systems, server applications, and standalone software. We report the
benchmarking results in terms of accuracy, robustness, and efficiency, which
are of practical importance when deploying automated log parsing in production.
We also share the success stories and lessons learned in an industrial
application at Huawei. We believe that our work could serve as the basis and
provide valuable guidance to future research and deployment of automated log
parsing.
