  The emergence of new wireless technologies together with the requirement of
massive connectivity results in several technical issues such as excessive
interference, high computational demand for signal processing, and lengthy
processing delays. In this work, we propose several beamforming techniques for
an uplink cell-free network with centralized, semi-distributed, and fully
distributed processing, all based on deep reinforcement learning (DRL). First,
we propose a fully centralized beamforming method that uses the deep
deterministic policy gradient algorithm (DDPG) with continuous space. We then
enhance this method by enabling distributed experience at access points (AP).
Indeed, we develop a beamforming scheme that uses the distributed
distributional deterministic policy gradients algorithm (D4PG) with the APs
representing the distributed agents. Finally, to decrease the computational
complexity, we propose a fully distributed beamforming scheme that divides the
beamforming computations among APs. The results show that the D4PG scheme with
distributed experience achieves the best performance irrespective of the
network size. Furthermore, the proposed distributed beamforming technique
performs better than the DDPG algorithm with centralized learning only for
small-scale networks. The performance superiority of the DDPG model becomes
more evident as the number of APs and/or users increases. Moreover, during the
operation stage, all DRL models demonstrate a significantly shorter processing
time than that of the conventional gradient descent (GD) solution.
