  The Open Radio Access Network (Open RAN) paradigm, and its reference
architecture proposed by the O-RAN Alliance, is paving the way toward open,
interoperable, observable and truly intelligent cellular networks. Crucial to
this evolution is Machine Learning (ML), which will play a pivotal role by
providing the necessary tools to realize the vision of self-organizing O-RAN
systems. However, to be actionable, ML algorithms need to demonstrate high
reliability, effectiveness in delivering high performance, and the ability to
adapt to varying network conditions, traffic demands and performance
requirements. To address these challenges, in this paper we propose a novel
Deep Reinforcement Learning (DRL) agent design for O-RAN applications that can
learn control policies under varying Service Level Agreement (SLAs) with
heterogeneous minimum performance requirements. We focus on the case of RAN
slicing and SLAs specifying maximum tolerable end-to-end latency levels. We use
the OpenRAN Gym open-source environment to train a DRL agent that can adapt to
varying SLAs and compare it against the state-of-the-art. We show that our
agent maintains a low SLA violation rate that is 8.3x and 14.4x lower than
approaches based on Deep Q- Learning (DQN) and Q-Learning while consuming
respectively 0.3x and 0.6x fewer resources without the need for re-training.
