  Massive multiple-input multiple-output (MIMO) avails of simple transceiver
design which can tackle many drawbacks of relay systems in terms of complicated
signal processing, latency, and noise amplification. However, the cost and
circuit complexity of having one radio frequency (RF) chain dedicated to each
antenna element are prohibitive in practice. In this paper, we address this
critical issue in amplify-and-forward (AF) relay systems using a hybrid analog
and digital (A/D) transceiver structure. More specifically, leveraging the
channel long-term properties, we design the analog beamformer which aims to
minimize the channel estimation error and remain invariant over a long
timescale. Then, the beamforming is completed by simple digital signal
processing, i.e., maximum ratio combining/maximum ratio transmission (MRC/MRT)
or zero-forcing (ZF) in the baseband domain. We present analytical bounds on
the achievable spectral efficiency taking into account the spatial correlation
and imperfect channel state information at the relay station. Our analytical
results reveal that the hybrid A/D structure with ZF digital processor exploits
spatial correlation and offers a higher spectral efficiency compared to the
hybrid A/D structure with MRC/MRT scheme. Our numerical results showcase that
the hybrid A/D beamforming design captures nearly 95% of the spectral
efficiency of a fully digital AF relaying topology even by removing half of the
RF chains. It is also shown that the hybrid A/D structure is robust to coarse
quantization, and even with 2-bit resolution, the system can achieve more than
93% of the spectral efficiency offered by the same hybrid A/D topology with
infinite resolution phase shifters.
