  In spite of the new opportunities brought about by the Open RAN, advances in
ML-based network automation have been slow, mainly because of the
unavailability of large-scale datasets and experimental testing infrastructure.
This slows down the development and widespread adoption of Deep Reinforcement
Learning (DRL) agents on real networks, delaying progress in intelligent and
autonomous RAN control. In this paper, we address these challenges by proposing
practical solutions and software pipelines for the design, training, testing,
and experimental evaluation of DRL-based closed-loop control in the Open RAN.
We introduce ColO-RAN, the first publicly-available large-scale O-RAN testing
framework with software-defined radios-in-the-loop. Building on the scale and
computational capabilities of the Colosseum wireless network emulator, ColO-RAN
enables ML research at scale using O-RAN components, programmable base
stations, and a "wireless data factory". Specifically, we design and develop
three exemplary xApps for DRL-based control of RAN slicing, scheduling and
online model training, and evaluate their performance on a cellular network
with 7 softwarized base stations and 42 users. Finally, we showcase the
portability of ColO-RAN to different platforms by deploying it on Arena, an
indoor programmable testbed. Extensive results from our first-of-its-kind
large-scale evaluation highlight the benefits and challenges of DRL-based
adaptive control. They also provide insights on the development of wireless DRL
pipelines, from data analysis to the design of DRL agents, and on the tradeoffs
associated to training on a live RAN. ColO-RAN and the collected large-scale
dataset will be made publicly available to the research community.
