  Non-parallel training is a difficult but essential task for DNN-based speech
enhancement methods, for the lack of adequate noisy and paired clean speech
corpus in many real scenarios. In this paper, we propose a novel adaptive
attention-in-attention CycleGAN (AIA-CycleGAN) for non-parallel speech
enhancement. In previous CycleGAN-based non-parallel speech enhancement
methods, the limited mapping ability of the generator may cause performance
degradation and insufficient feature learning. To alleviate this degradation,
we propose an integration of adaptive time-frequency attention (ATFA) and
adaptive hierarchical attention (AHA) to form an attention-in-attention (AIA)
module for more flexible feature learning during the mapping procedure. More
specifically, ATFA can capture the long-range temporal-spectral contextual
information for more effective feature representations, while AHA can flexibly
aggregate different AFTA's intermediate output feature maps by adaptive
attention weights depending on the global context. Numerous experimental
results demonstrate that the proposed approach achieves consistently more
superior performance over previous GAN-based and CycleGAN-based methods in
non-parallel training. Moreover, experiments in parallel training verify that
the proposed AIA-CycleGAN also outperforms most advanced GAN-based and Non-GAN
based speech enhancement approaches, especially in maintaining speech integrity
and reducing speech distortion.
