  We address the resource allocation challenges in NextG cellular radio access
networks (RAN), where heterogeneous user applications demand guarantees on
throughput and service regularity. We leverage the Whittle indexability
property to decompose the resource allocation problem, enabling the independent
computation of relative priorities for each user. By simply allocating
resources in decreasing order of these indices, we transform the combinatorial
complexity of resource allocation into a linear one. We propose Windex, a
lightweight approach for training neural networks to compute Whittle indices,
considering constraint violation, channel quality, and system load. Implemented
on a real-time RAN intelligent controller (RIC), our approach enables resource
allocation decision times of less than 20$\mu$s per user and efficiently
allocates resources in each 1ms scheduling time slot. Evaluation across
standardized 3GPP service classes demonstrates significant improvements in
service guarantees compared to existing schedulers, validated through
simulations and emulations with over-the-air channel traces on a 5G testbed.
