  Wav2vec2 has achieved success in applying Transformer architecture and
self-supervised learning to speech recognition. Recently, these have come to be
used not only for speech recognition but also for the entire speech processing.
This paper introduces an effective end-to-end speaker identification model
applied Transformer-based contextual model. We explored the relationship
between the hyper-parameters and the performance in order to discern the
structure of an effective model. Furthermore, we propose a pooling method,
Temporal Gate Pooling, with powerful learning ability for speaker
identification. We applied Conformer as encoder and BEST-RQ for pre-training
and conducted an evaluation utilizing the speaker identification of VoxCeleb1.
The proposed method has achieved an accuracy of 87.1% with 28.5M parameters,
demonstrating comparable precision to wav2vec2 with 317.7M parameters. Code is
available at https://github.com/HarunoriKawano/speaker-identification-with-tgp.
