  This paper proposed new hybrid, analog-digital, beamforming for a multiuser
millimeter wave (mm-wave) relay system. For this system, we consider a sum rate
maximization problem. The proposed hybrid beamforming is designed indirectly by
considering a sum mean square error (MSE) minimization problem while utilizing
the solution of digital beamforming. To this end, we assume that the digital
beamforming utilizes the well known block diagonalization (BD) approach. Under
this assumption, we solve our problem as follows: First, we formulate the sum
rate maximization problem as the minimization of the MSE between the received
signal of the hybrid and digital beamforming designs. Then, we design the
hybrid beamformings of the source, relay and each destination by leveraging
compressive sensing techniques. Simulation results confirm that the proposed
hybrid beamforming design achieves performance very close to that of the
digital one. Furthermore, we have examined the effects of the number of radio
frequency (RF) chains and paths together, and the accuracy of angle of arrival
(AoA) and angle of departure (AoD) estimators on the sum rate of the hybrid
beamforming designs. Computer simulations reveal that the total sum rate of the
hybrid beamforming increases when the number of RF chains and paths increase
(or the accuracy of the AoA (AoD) estimator improves).
