  This paper is focused on the finetuning of acoustic models for speaker
adaptation goals on a given gender. We pretrained the Transformer baseline
model on Librispeech-960 and conduct experiments with finetuning on the
gender-specific test subsets and. In general, we do not obtain essential WER
reduction by finetuning techniques by this approach. We achieved up to ~5%
lower word error rate on the male subset and 3% on the female subset if the
layers in the encoder and decoder are not frozen, but the tuning is started
from the last checkpoints. Moreover, we adapted our base model on the full L2
Arctic dataset of accented speech and fine-tuned it for particular speakers and
male and female genders separately. The models trained on the gender subsets
obtained 1-2% higher accuracy when compared to the model tuned on the whole L2
Arctic dataset. Finally, we tested the concatenation of the pretrained x-vector
voice embeddings and embeddings from a conventional encoder, but its gain in
accuracy is not significant.
