  While the open architecture, open interfaces, and integration of intelligence
within Open Radio Access Network technology hold the promise of transforming 5G
and 6G networks, they also introduce cybersecurity vulnerabilities that hinder
its widespread adoption. In this paper, we conduct a thorough system-level
investigation of cyber threats, with a specific focus on machine learning (ML)
intelligence components known as xApps within the O-RAN's near-real-time RAN
Intelligent Controller (near-RT RIC) platform. Our study begins by developing a
malicious xApp designed to execute adversarial attacks on two types of test
data - spectrograms and key performance metrics (KPMs), stored in the RIC
database within the near-RT RIC. To mitigate these threats, we utilize a
distillation technique that involves training a teacher model at a high softmax
temperature and transferring its knowledge to a student model trained at a
lower softmax temperature, which is deployed as the robust ML model within
xApp. We prototype an over-the-air LTE/5G O-RAN testbed to assess the impact of
these attacks and the effectiveness of the distillation defense technique by
leveraging an ML-based Interference Classification (InterClass) xApp as an
example. We examine two versions of InterClass xApp under distinct scenarios,
one based on Convolutional Neural Networks (CNNs) and another based on Deep
Neural Networks (DNNs) using spectrograms and KPMs as input data respectively.
Our findings reveal up to 100% and 96.3% degradation in the accuracy of both
the CNN and DNN models respectively resulting in a significant decline in
network performance under considered adversarial attacks. Under the strict
latency constraints of the near-RT RIC closed control loop, our analysis shows
that the distillation technique outperforms classical adversarial training by
achieving an accuracy of up to 98.3% for mitigating such attacks.
