  Logs have been an imperative resource to ensure the reliability and
continuity of many software systems, especially large-scale distributed
systems. They faithfully record runtime information to facilitate system
troubleshooting and behavior understanding. Due to the large scale and
complexity of modern software systems, the volume of logs has reached an
unprecedented level. Consequently, for log-based anomaly detection,
conventional manual inspection methods or even traditional machine
learning-based methods become impractical, which serve as a catalyst for the
rapid development of deep learning-based solutions. However, there is currently
a lack of rigorous comparison among the representative log-based anomaly
detectors that resort to neural networks. Moreover, the re-implementation
process demands non-trivial efforts, and bias can be easily introduced. To
better understand the characteristics of different anomaly detectors, in this
paper, we provide a comprehensive review and evaluation of five popular neural
networks used by six state-of-the-art methods. Particularly, four of the
selected methods are unsupervised, and the remaining two are supervised. These
methods are evaluated with two publicly available log datasets, which contain
nearly 16 million log messages and 0.4 million anomaly instances in total. We
believe our work can serve as a basis in this field and contribute to future
academic research and industrial applications.
