  The convolutional neural network (CNN) based approaches have shown great
success for speaker verification (SV) tasks, where modeling long temporal
context and reducing information loss of speaker characteristics are two
important challenges significantly affecting the verification performance.
Previous works have introduced dilated convolution and multi-scale aggregation
methods to address above challenges. However, such methods are also hard to
make full use of some valuable information, which make it difficult to
substantially improve the verification performance. To address above issues, we
construct a novel CNN-based architecture for SV, called RSKNet-MTSP, where a
residual selective kernel block (RSKBlock) and a multiple time-scale statistics
pooling (MTSP) module are first proposed. The RSKNet-MTSP can capture both long
temporal context and neighbouring information, and gather more
speaker-discriminative information from multi-scale features. In order to
design a portable model for real applications with limited resources, we then
present a lightweight version of RSKNet-MTSP, namely RSKNet-MTSP-L, which
employs a combination technique associating the depthwise separable
convolutions with low-rank factorization of weight matrices. Extensive
experiments are conducted on two public SV datasets, VoxCeleb and Speaker in
the Wild (SITW). The results demonstrate that 1) RSKNet-MTSP outperforms the
state-of-the-art deep embedding architectures by at least 9%-26% in all test
sets. 2) RSKNet-MTSP-L achieves competitive performance compared with baseline
models with 17%-39% less network parameters. The ablation experiments further
illustrate that our proposed approaches can achieve substantial improvement
over prior methods.
