  Massive MIMO has been the subject of intense interest in both academia and
industry for the past few years. 3GPP standardization for cellular systems have
adopted the principles of massive MIMO and categorized the use of large
rectangular planar arrays at the base station as full-dimension MIMO (FD-MIMO)
to operate in both TDD and FDD. Operating a large antenna array base station
requires the system to overcome several implementation challenges caused by
hardware impairments making practical solutions non-ideal and expensive to
deploy at scale. It is important to learn from existing challenges and
solutions in order to prepare for larger scale deployment for example with cell
free massive MIMO. Hence in this paper, we specifically study the phase and
amplitude instability due to RF impairments using measurements carried out in
the lab and in the field in a commercial LTE network. We investigate the effect
of phase and magnitude errors on the performance of FD-MIMO systems. We discuss
and characterize various sources creating these errors including time varying
phase drift from low cost local oscillator (LO) and internal temperature
variations affecting frequency response of RF chains. The minimum requirements
and tradeoffs of different LO architectures and calibration mechanisms for
practical cellular deployment are discussed. We then provide details of a novel
coherent LO distribution mechanism and related novel array calibration
mechanism that can be applied to both TDD and FDD systems. Measurement results
are provided to validate the performance of these methods used in a 2D
full-connected hybrid beamforming array architecture called High Definition
Active Antenna System (HDAAS). These results showcase the efficacy of the
proposed methods which can easily be extended to other array architectures
including sub-array hybrid beamforming and element-level digitization.
