  Intelligent reflecting surface (IRS) is envisioned to change the paradigm of
wireless communications from "adapting to wireless channels" to "changing
wireless channels". However, current IRS configuration schemes, consisting of
sub-channel estimation and passive beamforming in sequence, conform to the
conventional model-based design philosophies and are difficult to be realized
practically in the complex radio environment. To create the smart radio
environment, we propose a model-free design of IRS control that is independent
of the sub-channel channel state information (CSI) and requires the minimum
interaction between IRS and the wireless communication system. We firstly model
the control of IRS as a Markov decision process (MDP) and apply deep
reinforcement learning (DRL) to perform real-time coarse phase control of IRS.
Then, we apply extremum seeking control (ESC) as the fine phase control of IRS.
Finally, by updating the frame structure, we integrate DRL and ESC in the
model-free control of IRS to improve its adaptivity to different channel
dynamics. Numerical results show the superiority of our proposed joint DRL and
ESC scheme and verify its effectiveness in model-free IRS control without
sub-channel CSI.
