  One of the main challenges with generalized frequency division multiplexing
(GFDM) systems is prototype filter design. A poorly designed filter increases
inherent and out-of-band (OOB) interferences. In this paper, we introduce a
novel optimal prototype filter for GFDM systems that eliminates the negative
effects of intrinsic interferences. We introduce a complex-valued pulse shape
similar to a single-sideband (SSB) modulation scheme, which significantly
improves bandwidth efficiency. Specifically, we introduce an optimization
problem to design an optimal pulse shape filter to reduce all intrinsic
interference to zero. We derive analytical expressions to evaluate the bit
error rate (BER) of the system and show how the designed optimal prototype
filter outperforms its current counterparts.
