  Speaker diarization is a task to label an audio or video recording with the
identity of the speaker at each given time stamp. In this work, we propose a
novel machine learning framework to conduct real-time multi-speaker diarization
and recognition without prior registration and pretraining in a fully online
and reinforcement learning setting. Our framework combines embedding
extraction, clustering, and resegmentation into the same problem as an online
decision-making problem. We discuss practical considerations and advanced
techniques such as the offline reinforcement learning, semi-supervision, and
domain adaptation to address the challenges of limited training data and
out-of-distribution environments. Our approach considers speaker diarization as
a fully online learning problem of the speaker recognition task, where the
agent receives no pretraining from any training set before deployment, and
learns to detect speaker identity on the fly through reward feedbacks. The
paradigm of the reinforcement learning approach to speaker diarization presents
an adaptive, lightweight, and generalizable system that is useful for
multi-user teleconferences, where many people might come and go without
extensive pre-registration ahead of time. Lastly, we provide a desktop
application that uses our proposed approach as a proof of concept. To the best
of our knowledge, this is the first approach to apply a reinforcement learning
approach to the speaker diarization task.
