  Deep Learning has a wide application in the area of natural language
processing and image processing due to its strong ability of generalization. In
this paper, we propose a novel neural network structure for jointly optimizing
the transmitter and receiver in communication physical layer under fading
channels. We build up a convolutional autoencoder to simultaneously conduct the
role of modulation, equalization and demodulation. The proposed system is able
to design different mapping scheme from input bit sequences of arbitrary length
to constellation symbols according to different channel environments. The
simulation results show that the performance of neural network based system is
superior to traditional modulation and equalization methods in terms of time
complexity and bit error rate (BER) under fading channels. The proposed system
can also be combined with other coding techniques to further improve the
performance. Furthermore, the proposed system network is more robust to channel
variation than traditional communication methods.
