  Deep learning has dramatically improved the performance of sounds
recognition. However, learning acoustic models directly from the raw waveform
is still challenging. Current waveform-based models generally use time-domain
convolutional layers to extract features. The features extracted by single size
filters are insufficient for building discriminative representation of audios.
In this paper, we propose multi-scale convolution operation, which can get
better audio representation by improving the frequency resolution and learning
filters cross all frequency area. For leveraging the waveform-based features
and spectrogram-based features in a single model, we introduce two-phase method
to fuse the different features. Finally, we propose a novel end-to-end network
called WaveMsNet based on the multi-scale convolution operation and two-phase
method. On the environmental sounds classification datasets ESC-10 and ESC-50,
the classification accuracies of our WaveMsNet achieve 93.75% and 79.10%
respectively, which improve significantly from the previous methods.
