  The new frontier in cellular networks is harnessing the enormous spectrum
available at millimeter wave (mmWave) frequencies above 28 GHz. The challenging
radio propagation characteristics at these frequencies, and the use of highly
directional beamforming, lead to intermittent links between the base station
(BS) and the user equipment (UE). In this paper, we revisit the problem of cell
selection to maintain an acceptable level of service, despite the underlying
intermittent link connectivity typical of mmWave links. We propose a Markov
Decision Process (MDP) framework to study the properties and performance of our
proposed cell selection strategy, which jointly considers several factors such
as dynamic channel load and link quality. We use the Value Iteration Algorithm
(VIA) to solve the MDP, and obtain the optimal set of associations. We address
the multi user problem through a distributed iterative approach, in which each
UE characterizes the evolution of the system based on stationary channel
distribution and cell selection statistics of other UEs. Through simulation
results, we show that our proposed technique makes judicious handoff choices,
thereby providing a significant improvement in the overall network capacity.
Further, our technique reduces the total number of handoffs, thus lowering the
signaling overhead, while providing a higher quality of service to the UEs.
