  Deep neural networks have become an indispensable technique for audio source
separation (ASS). It was recently reported that a variant of CNN architecture
called MMDenseNet was successfully employed to solve the ASS problem of
estimating source amplitudes, and state-of-the-art results were obtained for
DSD100 dataset. To further enhance MMDenseNet, here we propose a novel
architecture that integrates long short-term memory (LSTM) in multiple scales
with skip connections to efficiently model long-term structures within an audio
context. The experimental results show that the proposed method outperforms
MMDenseNet, LSTM and a blend of the two networks. The number of parameters and
processing time of the proposed model are significantly less than those for
simple blending. Furthermore, the proposed method yields better results than
those obtained using ideal binary masks for a singing voice separation task.
