  In this paper, optimal filter design for generalized frequency-division
multiplexing (GFDM) is considered under two design criteria: rate maximization
and out-of-band (OOB) emission minimization. First, the problem of GFDM filter
optimization for rate maximization is formulated by expressing the transmission
rate of GFDM as a function of GFDM filter coefficients. It is shown that
Dirichlet filters are rate-optimal in additive white Gaussian noise (AWGN)
channels with no carrier frequency offset (CFO) under linear zero-forcing (ZF)
or minimum mean-square error (MMSE) receivers, but in general channels
perturbed by CFO a properly designed nontrivial GFDM filter can yield better
performance than Dirichlet filters by adjusting the subcarrier waveform to cope
with the channel-induced CFO. Next, the problem of GFDM filter design for OOB
emission minimization is formulated by expressing the power spectral density
(PSD) of the GFDM transmit signal as a function of GFDM filter coefficients,
and it is shown that the OOB emission can be reduced significantly by designing
the GFDM filter properly. Finally, joint design of GFDM filter and window for
the two design criteria is considered.
