  Log anomaly detection is a critical component in modern software system
security and maintenance, serving as a crucial support and basis for system
monitoring, operation, and troubleshooting. It aids operations personnel in
timely identification and resolution of issues. However, current methods in log
anomaly detection still face challenges such as underutilization of unlabeled
data, imbalance between normal and anomaly class data, and high rates of false
positives and false negatives, leading to insufficient effectiveness in anomaly
recognition. In this study, we propose a semi-supervised log anomaly detection
method named DQNLog, which integrates deep reinforcement learning to enhance
anomaly detection performance by leveraging a small amount of labeled data and
large-scale unlabeled data. To address issues of imbalanced data and
insufficient labeling, we design a state transition function biased towards
anomalies based on cosine similarity, aiming to capture semantic-similar
anomalies rather than favoring the majority class. To enhance the model's
capability in learning anomalies, we devise a joint reward function that
encourages the model to utilize labeled anomalies and explore unlabeled
anomalies, thereby reducing false positives and false negatives. Additionally,
to prevent the model from deviating from normal trajectories due to
misestimation, we introduce a regularization term in the loss function to
ensure the model retains prior knowledge during updates. We evaluate DQNLog on
three widely used datasets, demonstrating its ability to effectively utilize
large-scale unlabeled data and achieve promising results across all
experimental datasets.
