  With the formation of next generation wireless communication, a growing
number of new applications like internet of things, autonomous car, and drone
is crowding the unlicensed spectrum. Licensed network such as the long-term
evolution (LTE) also comes to the unlicensed spectrum for better providing
high-capacity contents with low cost. However, LTE was not designed for sharing
spectrum with others. A cooperation center for these networks is costly because
they possess heterogeneous properties and everyone can enter and leave the
spectrum unrestrictedly, so the design will be challenging. Since it is
infeasible to incorporate potentially infinite scenarios with one unified
design, an alternative solution is to let each network learn its own
coexistence policy. Previous solutions only work on fixed scenarios. In this
work a reinforcement learning algorithm is presented to cope with the
coexistence between Wi-Fi and LTE agents in 5 GHz unlicensed spectrum. The
coexistence problem was modeled as a decentralized partially observable Markov
decision process (Dec-POMDP) and Bayesian approach was adopted for policy
learning with nonparametric prior to accommodate the uncertainty of policy for
different agents. A fairness measure was introduced in the reward function to
encourage fair sharing between agents. The reinforcement learning was turned
into an optimization problem by transforming the value function as likelihood
and variational inference for posterior approximation. Simulation results
demonstrate that this algorithm can reach high value with compact policy
representations, and stay computationally efficient when applying to agent set.
