  Internet of things, supported by machine-to-machine (M2M) communications, is
one of the most important applications for future 6th generation (6G) systems.
A major challenge facing by 6G is enabling a massive number of M2M devices to
access networks in a timely manner. Therefore, this paper exploits the spatial
selectivity of massive multi-input multi-output (MIMO) to reduce the collision
issue when massive M2M devices initiate random access simultaneously. In
particular, a beam-based random access protocol is first proposed to make
efficient use of the limited uplink resources for massive M2M devices. To
address the non-uniform distribution of M2M devices in the space and time
dimensions, an Markov decision process (MDP) problem with the objective of
minimizing the average access delay is then formulated. Next, we present a
dynamic beam-based access scheme based on the double deep Q network (DDQN)
algorithm to solve the optimal policy. Finally, simulations are conducted to
demonstrate the effectiveness of the proposed scheme including the model
training and random access performance.
