  Reconfigurable intelligent surfaces (RIS) are expected to play an important
role in future wireless communication systems. These surfaces typically rely on
their reflection beamforming codebooks to reflect and focus the signal on the
target receivers. Prior work has mainly considered pre-defined RIS beamsteering
codebooks that do not adapt to the environment and hardware and lead to large
beam training overhead. In this work, a novel deep reinforcement learning based
framework is developed to efficiently construct the RIS reflection beam
codebook. This framework adopts a multi-level design approach that transfers
the learning between the multiple RIS subarrays, which speeds up the learning
convergence and highly reduces the computational complexity for large RIS
surfaces. The proposed approach is generic for co-located/distributed RIS
surfaces with arbitrary array geometries and with stationary/non-stationary
channels. Further, the developed solution does not require explicitly channel
knowledge and adapts the codebook beams to the surrounding environment, user
distribution, and hardware characteristics. Simulation results show that the
proposed learning framework can learn optimized interaction codebooks within
reasonable iterations. Besides, with only 6 beams, the learned codebook
outperforms a 256-beam DFT codebook, which significantly reduces the beam
training overhead.
