  Existing generative adversarial networks (GANs) for speech enhancement solely
rely on the convolution operation, which may obscure temporal dependencies
across the sequence input. To remedy this issue, we propose a self-attention
layer adapted from non-local attention, coupled with the convolutional and
deconvolutional layers of a speech enhancement GAN (SEGAN) using raw signal
input. Further, we empirically study the effect of placing the self-attention
layer at the (de)convolutional layers with varying layer indices as well as at
all of them when memory allows. Our experiments show that introducing
self-attention to SEGAN leads to consistent improvement across the objective
evaluation metrics of enhancement performance. Furthermore, applying at
different (de)convolutional layers does not significantly alter performance,
suggesting that it can be conveniently applied at the highest-level
(de)convolutional layer with the smallest memory overhead.
