  The highly heterogeneous ecosystem of NextG wireless communication systems
calls for novel networking paradigms where functionalities and operations can
be dynamically and optimally reconfigured in real time to adapt to changing
traffic conditions and satisfy stringent and diverse QoS demands. Open RAN
technologies, and specifically those being standardized by the O-RAN Alliance,
make it possible to integrate network intelligence into the once monolithic RAN
via intelligent applications, namely, xApps and rApps. These applications
enable flexible control of the network resources and functionalities, network
management, and orchestration through data-driven intelligent control loops.
Recent work has showed how DRL is effective in dynamically controlling O-RAN
systems. However, how to design these solutions in a way that manages
heterogeneous optimization goals and prevents unfair resource allocation is
still an open challenge, with the logic within DRL agents often considered as a
black box. In this paper, we introduce PandORA, a framework to automatically
design and train DRL agents for Open RAN applications, package them as xApps
and evaluate them in the Colosseum wireless network emulator. We benchmark $23$
xApps that embed DRL agents trained using different architectures, reward
design, action spaces, and decision-making timescales, and with the ability to
hierarchically control different network parameters. We test these agents on
the Colosseum testbed under diverse traffic and channel conditions, in static
and mobile setups. Our experimental results indicate how suitable fine-tuning
of the RAN control timers, as well as proper selection of reward designs and
DRL architectures can boost network performance according to the network
conditions and demand. Notably, finer decision-making granularities can improve
mMTC's performance by ~56% and even increase eMBB Throughput by ~99%.
