  Most enterprise applications use logging as a mechanism to diagnose
anomalies, which could help with reducing system downtime. Anomaly detection
using software execution logs has been explored in several prior studies, using
both classical and deep neural network-based machine learning models. In recent
years, the research has largely focused in using variations of sequence-based
deep neural networks (e.g., Long-Short Term Memory and Transformer-based
models) for log-based anomaly detection on open-source data. However, they have
not been applied in industrial datasets, as often. In addition, the studied
open-source datasets are typically very large in size with logging statements
that do not change much over time, which may not be the case with a dataset
from an industrial service that is relatively new. In this paper, we evaluate
several state-of-the-art anomaly detection models on an industrial dataset from
our research partner, which is much smaller and loosely structured than most
large scale open-source benchmark datasets. Results show that while all models
are capable of detecting anomalies, certain models are better suited for
less-structured datasets. We also see that model effectiveness changes when a
common data leak associated with a random train-test split in some prior work
is removed. A qualitative study of the defects' characteristics identified by
the developers on the industrial dataset further shows strengths and weaknesses
of the models in detecting different types of anomalies. Finally, we explore
the effect of limited training data by gradually increasing the training set
size, to evaluate if the model effectiveness does depend on the training set
size.
