  The exponential increase of wireless devices with highly demanding services
such as streaming video, gaming and others has imposed several challenges to
Wireless Local Area Networks (WLANs). In the context of Wi-Fi, IEEE 802.11ax
brings high-data rates in dense user deployments. Additionally, it comes with
new flexible features in the physical layer as dynamic Clear-Channel-Assessment
(CCA) threshold with the goal of improving spatial reuse (SR) in response to
radio spectrum scarcity in dense scenarios. In this paper, we formulate the
Transmission Power (TP) and CCA configuration problem with an objective of
maximizing fairness and minimizing station starvation. We present four main
contributions into distributed SR optimization using Multi-Agent Multi-Armed
Bandits (MAMABs). First, we propose to reduce the action space given the large
cardinality of action combination of TP and CCA threshold values per Access
Point (AP). Second, we present two deep Multi-Agent Contextual MABs (MA-CMABs),
named Sample Average Uncertainty (SAU)-Coop and SAU-NonCoop as cooperative and
non-cooperative versions to improve SR. In addition, we present an analysis
whether cooperation is beneficial using MA-MABs solutions based on the
e-greedy, Upper Bound Confidence (UCB) and Thompson techniques. Finally, we
propose a deep reinforcement transfer learning technique to improve
adaptability in dynamic environments. Simulation results show that cooperation
via SAU-Coop algorithm contributes to an improvement of 14.7% in cumulative
throughput, and 32.5% improvement of PLR when compared with no cooperation
approaches. Finally, under dynamic scenarios, transfer learning contributes to
mitigation of service drops for at least 60% of the total of users.
