  In this paper, a new matrix-based characterization of
generalized-frequency-division-multiplexing (GFDM) transmitter matrices is
proposed, as opposed to traditional vector-based characterization with
prototype filters. The characterization facilitates deriving properties of GFDM
(transmitter) matrices, including conditions for GFDM matrices being
nonsingular and unitary, respectively. Using the new characterization, the
necessary and sufficient conditions for the existence of a form of
low-complexity implementation for a minimum mean square error (MMSE) receiver
are derived. Such an implementation exists under multipath channels if the GFDM
transmitter matrix is selected to be unitary. For cases where this
implementation does not exist, a low-complexity suboptimal MMSE receiver is
proposed, with its performance approximating that of an MMSE receiver. The new
characterization also enables derivations of optimal prototype filters in terms
of minimizing receiver mean square error (MSE). They are found to correspond to
the use of unitary GFDM matrices under many scenarios. The use of such optimal
filters in GFDM systems does not cause the problem of noise enhancement,
thereby demonstrating the same MSE performance as orthogonal frequency division
multiplexing. Moreover, we find that GFDM matrices with a size of power of two
are verified to exist in the class of unitary GFDM matrices. Finally, while the
out-of-band (OOB) radiation performance of systems using a unitary GFDM matrix
is not optimal in general, it is shown that the OOB radiation can be
satisfactorily low if parameters in the new characterization are carefully
chosen.
