  The nested array, which is implemented by nonuniform antenna placement, is an
effective approach to achieve O(N^2) degrees of freedom (DoF) with an antenna
array of N antennas. Such DoF refers to the number of directions of incoming
signals that can be resolved. With the increased number of DoF, an important
application of nested array is to nullify the interference signals from
multiple directions. In this paper, we apply nested array in a massive MIMO
heterogeneous network (HetNet) for interference management. With nested array
based interference nulling, each base station (BS) can nullify a certain number
of interference signals. A key design issue is to select the interference
sources to be nullified at each BS. We formulate this problem as an integer
programming problem. The objective is to maximize the sum rate of all users,
subject to BS DoF constraints. We propose an approximation scheme to solve this
problem and derive a performance upper bound. Simulation results show that the
proposed scheme effectively improves the sum rate and achieves a near optimal
performance.
