  Cross-speaker style transfer aims to extract the speech style of the given
reference speech, which can be reproduced in the timbre of arbitrary target
speakers. Existing methods on this topic have explored utilizing
utterance-level style labels to perform style transfer via either global or
local scale style representations. However, audiobook datasets are typically
characterized by both the local prosody and global genre, and are rarely
accompanied by utterance-level style labels. Thus, properly transferring the
reading style across different speakers remains a challenging task. This paper
aims to introduce a chunk-wise multi-scale cross-speaker style model to capture
both the global genre and the local prosody in audiobook speeches. Moreover, by
disentangling speaker timbre and style with the proposed switchable adversarial
classifiers, the extracted reading style is made adaptable to the timbre of
different speakers. Experiment results confirm that the model manages to
transfer a given reading style to new target speakers. With the support of
local prosody and global genre type predictor, the potentiality of the proposed
method in multi-speaker audiobook generation is further revealed.
