  We propose a learning-based method for the joint design of a transmit and
receive filter, the constellation geometry and associated bit labeling, as well
as a neural network (NN)-based detector. The method maximizes an achievable
information rate, while simultaneously satisfying constraints on the adjacent
channel leakage ratio (ACLR) and peak-to-average power ratio (PAPR). This
allows control of the tradeoff between spectral containment, peak power, and
communication rate. Evaluation on an additive white Gaussian noise (AWGN)
channel shows significant reduction of ACLR and PAPR compared to a conventional
baseline relying on quadrature amplitude modulation (QAM) and
root-raised-cosine (RRC), without significant loss of information rate. When
considering a 3rd Generation Partnership Project (3GPP) multipath channel, the
learned waveform and neural receiver enable competitive or higher rates than an
orthogonal frequency division multiplexing (OFDM) baseline, while reducing the
ACLR by 10 dB and the PAPR by 2 dB. The proposed method incurs no additional
complexity on the transmitter side and might be an attractive tool for waveform
design of beyond-5G systems.
