  In evolving cyber landscapes, the detection of malicious URLs calls for
cooperation and knowledge sharing across domains. However, collaboration is
often hindered by concerns over privacy and business sensitivities. Federated
learning addresses these issues by enabling multi-clients collaboration without
direct data exchange. Unfortunately, if highly expressive Transformer models
are used, clients may face intolerable computational burdens, and the exchange
of weights could quickly deplete network bandwidth. In this paper, we propose
Fed-urlBERT, a federated URL pre-trained model designed to address both privacy
concerns and the need for cross-domain collaboration in cybersecurity.
Fed-urlBERT leverages split learning to divide the pre-training model into
client and server part, so that the client part takes up less extensive
computation resources and bandwidth. Our appraoch achieves performance
comparable to centralized model under both independently and identically
distributed (IID) and two non-IID data scenarios. Significantly, our federated
model shows about an 7% decrease in the FPR compared to the centralized model.
Additionally, we implement an adaptive local aggregation strategy that
mitigates heterogeneity among clients, demonstrating promising performance
improvements. Overall, our study validates the applicability of the proposed
Transformer federated learning for URL threat analysis, establishing a
foundation for real-world collaborative cybersecurity efforts. The source code
is accessible at https://github.com/Davidup1/FedURLBERT.
