  In this paper, we propose a novel deep unsupervised learning-based approach
that jointly optimizes antenna selection and hybrid beamforming to improve the
hardware and spectral efficiencies of massive multiple-input-multiple-output
(MIMO) downlink systems. By employing ResNet to extract features from the
channel matrices, two neural networks, i.e., the antenna selection network
(ASNet) and the hybrid beamforming network (BFNet), are respectively proposed
for dynamic antenna selection and hybrid beamformer design. Furthermore, a deep
probabilistic subsampling trick and a specially designed quantization function
are respectively developed for ASNet and BFNet to preserve the
differentiability while embedding discrete constraints into the network
structures. With the aid of a flexibly designed loss function, ASNet and BFNet
are jointly trained in a phased unsupervised way, which avoids the prohibitive
computational cost of acquiring training labels in supervised learning.
Simulation results demonstrate the advantage of the proposed approach over
conventional optimization-based algorithms in terms of both the achieved rate
and the computational complexity.
