  Logs provide users with useful insights to help with a variety of development
and operations tasks. The problem is that logs are often unstructured, making
their analysis a complex task. This is mainly due to the lack of guidelines and
best practices for logging, combined with a large number of logging libraries
at the disposal of software developers. There exist studies that aim to parse
automatically large logs. The main objective is to extract templates from
samples of log data that are used to recognize future logs. In this paper, we
propose AWSOM-LP, a powerful log parsing and abstraction tool, which is highly
accurate, stable, and efficient. AWSOM-LP is built on the idea of applying
pattern recognition and frequency analysis. First, log events are organized
into patterns using a simple text processing method. Frequency analysis is then
applied locally to instances of the same group to identify static and dynamic
content of log events. When applied to 16 log datasets of the the LogPai
project, AWSOM-LP achieves an average grouping accuracy of 93.5%, which
outperforms the accuracy of five leading log parsing tools namely, Logram,
Lenma, Drain, IPLoM and AEL. Additionally, AWSOM-LP can generate more than 80%
of the final log templates from 10% to 50% of the entire log dataset and can
parse up to a million log events in an average time of 5 minutes. AWSOM-LP is
available online as an open source. It can be used by practitioners and
researchers to parse effectively and efficiently large log files so as to
support log analysis tasks.
