  A multitude of toxic online behaviors, ranging from network attacks to
anonymous traffic and spam, have severely disrupted the smooth operation of
networks. Due to the inherent sender-receiver nature of network behaviors,
graph-based frameworks are commonly used for detecting anomalous behaviors.
However, in real-world scenarios, the boundary between normal and anomalous
behaviors tends to be ambiguous. The local heterophily of graphs interferes
with the detection, and existing methods based on nodes or edges introduce
unwanted noise into representation results, thereby impacting the effectiveness
of detection. To address these issues, we propose PhoGAD, a graph-based anomaly
detection framework. PhoGAD leverages persistent homology optimization to
clarify behavioral boundaries. Building upon this, the weights of adjacent
edges are designed to mitigate the effects of local heterophily. Subsequently,
to tackle the noise problem, we conduct a formal analysis and propose a
disentangled representation-based explicit embedding method, ultimately
achieving anomaly behavior detection. Experiments on intrusion, traffic, and
spam datasets verify that PhoGAD has surpassed the performance of
state-of-the-art (SOTA) frameworks in detection efficacy. Notably, PhoGAD
demonstrates robust detection even with diminished anomaly proportions,
highlighting its applicability to real-world scenarios. The analysis of
persistent homology demonstrates its effectiveness in capturing the topological
structure formed by normal edge features. Additionally, ablation experiments
validate the effectiveness of the innovative mechanisms integrated within
PhoGAD.
